FeDXL: Provable Federated Learning for Deep X-Risk Optimization
Jul 24, 2023
Sprecher:innen
Zhishuai Guo
Sprecher:in · 0 Follower:innen
Tianbao Yang
Sprecher:in · 0 Follower:innen
Über
In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of 𝔼_𝐳∼𝒮_1 f(𝔼_𝐳'∼𝒮_2ℓ(𝐰; 𝐳, 𝐳')), where two sets of data 𝒮_1, 𝒮_2 are distributed over multiple machines, ℓ(·; ·,·) is a pairwise loss that only depends on the prediction outputs of the input data pairs (𝐳, 𝐳').This problem has important applications in machine learning, e.g., AUROC maximization with a pairwise loss, and partial AUROC maximization with a compositional loss. The challenges for designing an FL algorithm for X-risks lie in the non-decomposability of the objective over multiple machines and the interdependency between different machines. To this end, we propose an active-passive decomposition framework that decouples the gradient's components with two types, namely active parts and passive parts, where the active parts depend on local data that are computed with the local model and the passive parts depend on other machines that are communicated/computed based on historical models and samples. Under this framework, we design two FL algorithms (FeDXL) for handling linear and nonlinear f, respectively, based on federated averaging and merging and develop a novel theoretical analysis to combat the latency of the passive parts and the interdependency between the local model parameters and the involved data for computing local gradient estimators. We establish both iteration and communication complexities and show that using the historical samples and models for computing the passive parts do not degrade the complexities. We conduct empirical studies of FeDXL for deep AUROC and partial AUROC maximization, and demonstrate their performance compared with several baselines.In this paper, we tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing FL algorithms are applicable. In particular, the objective has the form of 𝔼_𝐳∼𝒮_1 f(𝔼_𝐳'∼𝒮_2ℓ(𝐰; 𝐳, 𝐳')), where two sets of data 𝒮_1, 𝒮_2 are distributed over multiple machines, ℓ(·; ·,·) is a pairwise loss that only depends on the prediction outputs of the input data pairs (𝐳, 𝐳').This problem has important applications in machine learning, e.g., AUROC maximiza…
Gefällt euch das Format? Vertraut auf SlidesLive, um euer nächstes Event festzuhalten!
Professionelle Aufzeichnung und Livestreaming – weltweit.
Freigeben
Empfohlene Videos
Präsentationen, deren Thema, Kategorie oder Sprecher:in ähnlich sind
Learning Shared Safety Constraints from Multi-task Demonstrations
Später ansehen
Gokul Swamy, …
Counterfactual reasoning: From minds to machines to practical applications
Später ansehen
Ruth Byrne, …
Behavior Contrastive learning for Unsupervised Skill Discovery
Später ansehen
Rushuai Yang, …
Towards Robust Graph Incremental Learning on Evolving Graphs
Später ansehen
Junwei Su, …
