Exponential Graphs are Provably Efficient for Decentralized Deep Training
Dec 6, 2021
Speakers
About
Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an under-studied topic in decentralized optimization.In this paper, we study so-called exponential graphs where every node is connected to O(log(n)) neighbors and n is the total number of nodes. This work proves such graphs can lead to both fast communication and effective averaging simultaneously. We also discover that a sequence of log(n) one-peer exponential graphs, in which each node communicates to one single neighbor per iteration, can together achieve exact averaging. This favorable property enables one-peer exponential graph to average as effective as its static counterpart but communicates more efficiently. We apply these exponential graphs in decentralized (momentum) SGD to obtain the state-of-the-art balance between per-iteration communication and iteration complexity among all commonly-used topologies. Experimental results on a variety of tasks and models demonstrate that decentralized (momentum) SGD over exponential graphs promises both fast and high-quality training.Decentralized SGD is an emerging training method for deep learning known for its much less (thus faster) communication per iteration, which relaxes the averaging step in parallel SGD to inexact averaging. The less exact the averaging is, however, the more the total iterations the training needs to take. Therefore, the key to making decentralized SGD efficient is to realize nearly-exact averaging using little communication. This requires a skillful choice of communication topology, which is an un…
Organizer
NeurIPS 2021
Account · 1.9k followers
About NeurIPS 2021
Neural Information Processing Systems (NeurIPS) is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers. Following the conference, there are workshops which provide a less formal setting.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Open Rule Induction
Watch later
Wanyun Cui, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Fair Sparse Regression with Clustering: An Invex Relaxation for a Combinatorial Problem
Watch later
Adarsh Barik, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Variational Optimistic Bayesian Sampling
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Meta Learning Backpropagation And Improving It
Watch later
Louis Kirsch, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Boosting With Multiple Sources
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Boost Neural Networks by Checkpoints
Watch later
Feng Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%