SLAMB: Accelerated Large Batch Training with Sparse Communication
Jul 24, 2023
Speakers
Wenxuan Zhang
Speaker · 0 followers
Jiawei Fei
Speaker · 0 followers
TingWen Xie
Speaker · 0 followers
Yuchen Xie
Speaker · 0 followers
Mohamed Elhoseiny
Speaker · 1 follower
Panos Kalnis
Speaker · 0 followers
About
Distributed training of large deep neural networks requires frequent exchange of massive data between machines, thus communication efficiency is a major concern. Existing compressed communication methods are either not compatible with large batch optimization algorithms, or do not provide sufficient speedup in large scale.In this paper, we combine sparsification-based gradient compression with the layer-wise adaptive moments optimizer for large batch training (LAMB). We propose SLAMB, a novel communication-efficient optimizer that supports large batch sizes and scales to thousands of GPUs. SLAMB employs momentum masking, local error compensation, and element-wise adaptive rescaling to achieve accurate layer-wise weight updates, which translates to fast convergence for very large batches. Our empirical results show that, compared to the state-of-the-art, SLAMB transmits half the amount of data in large-batch BERT pre-training, without sacrificing accuracy. Moreover, SLAMB achieves excellent scalability in large computing infrastructures. For instance, SLAMB with 128 GPUs reduces the training time of Swin Transformer pre-training on ImageNet to 5.35 hours, which is 2 hours faster than the state-of-the-art.At the extreme, we trained BERT-XL (2.8B parameters) on 1,024 NVIDIA A100 GPUs, where SLAMB achieved 90% scaling efficiency.Distributed training of large deep neural networks requires frequent exchange of massive data between machines, thus communication efficiency is a major concern. Existing compressed communication methods are either not compatible with large batch optimization algorithms, or do not provide sufficient speedup in large scale.In this paper, we combine sparsification-based gradient compression with the layer-wise adaptive moments optimizer for large batch training (LAMB). We propose SLAMB, a novel co…
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Model Transferability with Responsive Decision Subjects
Watch later
Yatong Chen, …
Towards Robust Graph Incremental Learning on Evolving Graphs
Watch later
Junwei Su, …
Fair yet Asymptotically Equal Collaborative Learning
Watch later
Brauer's Group Equivariant Neural Networks
Watch later
Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
Watch later
Yuhang Ran, …
