Lowering the Pre-training Tax for Gradient-Based Subset Training
Jul 24, 2023
Speakers
Zhangyang Wang
Speaker · 1 follower
Vijay Chidambaram
Speaker · 0 followers
Aditya Akella
Speaker · 0 followers
About
Training data and model sizes are increasing exponentially.One way to reduce training time and resources is to train with a carefully selected subset of the full dataset.Prior work uses the gradient signals obtained during a warm-up or “pre-training" phase over the full dataset, for determining the core subset;if the pre-training phase is too small, the gradients obtained are chaotic and unreliable. As a result, the pre-training phase itself incurs significant time/resource overhead;prior work has not gone beyond hyperparameter search to reduce pre-training time. Our work explicitly aims to reduce this pre-training tax in gradient-based subset training. We develop a principled, scalable approach for pre-training in a distributed setup. Our approach is lightweightand minimizes communication between distributed worker nodes. It is the first to utilize the concept of model-soup based distributed training at initialization.The key idea is to minimally train an ensemble of models on small, disjointed subsets of the data;we further employ data-driven sparsity and data augmentation for local worker training to boost ensemble diversity.The centralized model, obtained at the end of pre-training by merging the per-worker models, is found to offer stabilized gradient signals to select subsets, on which the model is further trained. We have validated the effectiveness of our method through extensive experiments on CIFAR-10/100, and ImageNet, using ResNet and WideResNet models. For example, our approach is shown to achieve 15.4× pre-training speedup and 2.8× end-to-end speedup on CIFAR10 and ResNet18 without loss of accuracy. Codes will be fully released.Training data and model sizes are increasing exponentially.One way to reduce training time and resources is to train with a carefully selected subset of the full dataset.Prior work uses the gradient signals obtained during a warm-up or “pre-training" phase over the full dataset, for determining the core subset;if the pre-training phase is too small, the gradients obtained are chaotic and unreliable. As a result, the pre-training phase itself incurs significant time/resource overhead;prior work h…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Robust and Scalable Bayesian Online Changepoint Detection
Watch later
On the Within-Group Fairness of Screening Classifiers
Watch later
GRIL: a 2-parameter Persistence Based Vectorization for Machine Learning
Watch later
Cheng Xin, …
Towards Speech ChatGPT: Recent Progress of Speech Language Model
Watch later
Panel Discussion: “Challenges and lessons learned in deploying Generative AI”
Watch later
Deep Ganguli, …