Weighted Mutual Learning with Diversity-Driven Model Compression
Nov 28, 2022
Speakers
Miao Zhang
Speaker · 0 followers
David Campos
Speaker · 0 followers
Chenjuan Guo
Speaker · 0 followers
About
Online distillation attracts attention from the community as it simplifies the traditional two-stage knowledge distillation process into a single stage. Online distillation collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory consumption and diversity among peers are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML) for online distillation. First, at the base of a hierarchical structure where peers share different parts, we leverage the structured network pruning to generate diversified peer models and reduce the memory requirements. Second, rather than taking the average of peers, this paper, for the first time, leverages a bi-level formulation to estimate the relative importance of peers with a close-form, to further boost the effectiveness of the distillation from each other. Extensive experiments show the generalization of the proposed framework, which outperforms existing online distillation methods on a variety of deep neural networks. More interesting, as a byproduct, WML produces a series of pruned models under different model sizes in a single run, which also achieves competitive results compared with existing channel pruning methods.Online distillation attracts attention from the community as it simplifies the traditional two-stage knowledge distillation process into a single stage. Online distillation collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory consumption and diversity among peers are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework…
Organizer
NeurIPS 2022
Account · 954 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
MOMA-LRG: Language-Refined Graphs for Multi-Object Multi-Actor Activity Parsing
Watch later
Zelun Luo, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Optimal Transport of Classifiers to Fairness
Watch later
Maarten Buyl, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Featured Papers Panels 4B
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Investigating Reproducibility and Double Descent from the Decision Boundary Perspective
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space](https://ma.slideslive.com/library/presentations/38992707/thumbnail/re-a-clusterbased-approach-for-improving-isotropy-in-contextual-embedding-space_jaiBg4_small.jpg)
[Re] A Cluster-based Approach for Improving Isotropy in Contextual Embedding Space
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%