Get More at Once: Alternating Sparse Training with Gradient Correction
Nov 28, 2022
Speakers
About
Recently, a new trend of exploring training sparsity has emerged, which remove parameters during training, leading to both training and inference efficiency improvement. This line of works primarily aims to obtain a single sparse model under a pre-defined large sparsity ratio. It leads to a static/fixed sparse inference model that is not capable of adjusting or re-configuring its computation complexity (i.e., inference structure, latency) after training for real-world varying and dynamic hardware resource availability. To enable such run-time or post-training network morphing, the concept of `dynamic inference' or `training-once-for-all' has been proposed to train a single network consisting of multiple sub-nets once, but each sub-net could perform the same inference function with different computing complexity. However, the traditional dynamic inference training method requires a joint training scheme with multi-objective optimization, which suffers from very large training overhead. In this work, for the first time, we propose a novel alternating sparse training (AST) scheme to train multiple sparse sub-nets for dynamic inference without extra training cost compared to the case of training a single sparse model from scratch. Furthermore, to mitigate the interference of weight update among sub-nets, we propose gradient correction within the inner-group iterations to reduce their weight update interference. We validate the proposed AST on multiple datasets against state-of-the-art sparse training method, which shows that AST achieves similar or better accuracy, but only needs to train once to get multiple sparse sub-nets with different sparsity ratios. More importantly, comparing with the traditional joint training based dynamic inference training methodology, the large training overhead is completely eliminated without affecting the accuracy of each sub-net.Recently, a new trend of exploring training sparsity has emerged, which remove parameters during training, leading to both training and inference efficiency improvement. This line of works primarily aims to obtain a single sparse model under a pre-defined large sparsity ratio. It leads to a static/fixed sparse inference model that is not capable of adjusting or re-configuring its computation complexity (i.e., inference structure, latency) after training for real-world varying and dynamic hardwar…
Organizer
NeurIPS 2022
Account · 958 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Learning Enhanced Representation for Tabular Data via Neighborhood Propagation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Task Discovery: Finding the Tasks that Neural Networks Generalize on
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Structure-Aware Image Segmentation with Homotopy Warping
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Communication Efficient Distributed Learning for Kernelized Contextual Bandits
Watch later
Chuanhao Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A unified information-theoretic model of EEG signatures of human language processing
Watch later
Jiaxuan Li, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Unsupervised Learning under Latent Label Shift
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%