Towards Understanding Ensemble Distillation in Federated Learning
Jul 24, 2023
Speakers
Sejun Park
Speaker · 0 followers
Kihun Hong
Speaker · 0 followers
Ganguk Hwang
Speaker · 0 followers
About
Federated Learning (FL) is a collaborative machine learning paradigm for data privacy preservation.Recently, a knowledge distillation (KD) based information sharing approach in FL, which conducts ensemble distillation on an unlabeled public dataset, has been proposed. However, despite its experimental success and usefulness, the theoretical analysis of the KD based approach has not been satisfactorily conducted. In this work, we build a theoretical foundation of the ensemble distillation framework in federated learning from the perspective of kernel ridge regression (KRR). In this end, we propose a KD based FL algorithm for KRR models which is related with some existing KD based FL algorithms, and analyze our algorithm theoretically. We show that our algorithm makes local prediction models as much powerful as the centralized KRR model (which is a KRR model trained by all of local datasets) in terms of the convergence rate of the generalization error if the unlabeled public dataset is sufficiently large. We also provide experimental results to verify our theoretical results on ensemble distillation in federated learning.Federated Learning (FL) is a collaborative machine learning paradigm for data privacy preservation.Recently, a knowledge distillation (KD) based information sharing approach in FL, which conducts ensemble distillation on an unlabeled public dataset, has been proposed. However, despite its experimental success and usefulness, the theoretical analysis of the KD based approach has not been satisfactorily conducted. In this work, we build a theoretical foundation of the ensemble distillation framewo…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Density of Reachable States for Safe Autonomous Motion Planning
Watch later
Grounding Language Models to Images for Multimodal Inputs and Outputs
Watch later
Jing Yu Koh, …
Provably Correct Physics-Informed Neural Networks
Watch later
Visual Storytelling with Generative Models of Video
Watch later
Subequivariant Graph Reinforcement Learning in 3D Environments
Watch later
Runfa Chen, …
