A Communication-Efficient Distributed Gradient Clipping Algorithm for Training Deep Neural Networks
Nov 28, 2022
Speakers
Mingrui Liu
Speaker · 0 followers
Zhenxun Zhuang
Speaker · 0 followers
Yunwen Lei
Speaker · 0 followers
Chunyang Liao
Speaker · 0 followers
About
In distributed training of deep neural networks, people usually run Stochastic Gradient Descent (SGD) or its variants on each machine and communicate with other machines periodically. However, SGD might converge slowly in training some deep neural networks (e.g., RNN, LSTM) because of the exploding gradient issue. Gradient clipping is usually employed to address this issue in the single machine setting, but exploring this technique in the distributed setting is still in its infancy: it remains mysterious whether the gradient clipping scheme can take advantage of multiple machines to enjoy parallel speedup. The main technical difficulty lies in dealing with nonconvex loss function, non-Lipschitz continuous gradient, and skipping communication rounds simultaneously. In this paper, we explore a relaxed-smoothness assumption of the loss landscape which LSTM was shown to satisfy in previous works, and design a communication-efficient gradient clipping algorithm. This algorithm can be run on multiple machines, where each machine employs a gradient clipping scheme and communicate with other machines after multiple steps of gradient-based updates. Our algorithm is proved to have O(1/Nϵ^4) iteration complexity and O(1/ϵ^3) communication complexity for finding an ϵ-stationary point in the homogeneous data setting, where N is the number of machines. This indicates that our algorithm enjoys linear speedup and reduced communication rounds. Our proof relies on novel analysis techniques of estimating truncated random variables, which we believe are of independent interest. Our experiments on several benchmark datasets and various scenarios demonstrate that our algorithm indeed exhibits fast convergence speed in practice and thus validates our theory.In distributed training of deep neural networks, people usually run Stochastic Gradient Descent (SGD) or its variants on each machine and communicate with other machines periodically. However, SGD might converge slowly in training some deep neural networks (e.g., RNN, LSTM) because of the exploding gradient issue. Gradient clipping is usually employed to address this issue in the single machine setting, but exploring this technique in the distributed setting is still in its infancy: it remains m…
Organizer
NeurIPS 2022
Account · 952 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Scalable Distributional Robustness in a Class of Non-Convex Optimization with Guarantees
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Distributional Convergence of the Sliced Wasserstein Process
Watch later
Jiaqi Xi, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning
Watch later
Seungjae Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Embodied Scene-aware Human Pose Estimation
Watch later
Zhengyi Luo, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Impact of realistic properties of the point spread function on classification tasks to reveal a possible distribution shift
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Driving SMARTS: Autonomous Driving Competition
Watch later
Tianpei Yang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%