Preservation of the Global Knowledge by Not-True Distillation in Federated Learning
Nov 28, 2022
Speakers
Minchan Jeong
Speaker · 0 followers
Yongjin Shin
Speaker · 0 followers
Sangmin Bae
Speaker · 0 followers
Se-Young Yun
Speaker · 0 followers
About
In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces forgetting the knowledge outside of the local distribution. Based on our findings, we hypothesize that tackling down forgetting will relieve the data heterogeneity problem. To this end, we propose a novel and effective algorithm, Federated Not-True Distillation (FedNTD), which preserves the global perspective on locally available data only for the not-true classes. In the experiments, FedNTD shows state-of-the-art performance on various setups without compromising data privacy or incurring additional communication costs.In federated learning, a strong global model is collaboratively learned by aggregating clients' locally trained models. Although this precludes the need to access clients' data directly, the global model's convergence often suffers from data heterogeneity. This study starts from an analogy to continual learning and suggests that forgetting could be the bottleneck of federated learning. We observe that the global model forgets the knowledge from previous rounds, and the local training induces for…
Organizer
NeurIPS 2022
Account · 961 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Simulations for Open Science Token Communities: Designing the Knowledge Commons
Watch later
Jakub Smékal, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Distributed Inverse Constrained Reinforcement Learning (D-ICRL) for Multi-agent Systems (MASs)
Watch later
Shicheng Liu, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Best paper announcement
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Model-based Trajectory Stitching for Improved Off-line Reinforcement Learning
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
SAGDA: Achieving 𝒪(ϵ^-2) Communication Complexity in Federated Min-Max Learning
Watch later
Haibo Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Efficient Planning in a Compact Latent Action Space
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%