Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
Nov 28, 2022
Speakers
Bokun Wang
Speaker · 0 followers
Mher Safaryan
Speaker · 0 followers
Peter Richtárik
Speaker · 0 followers
About
To address the high communication costs of distributed machine learning, a large body of work has been devoted in recent years to designing various compression strategies, such as sparsification and quantization, and optimization algorithms capable of using them. Recently, Safaryan et al. [2021] pioneered a dramatically different compression design approach: they first use the local training data to form local smoothness matrices and then propose to design a compressor capable of exploiting the smoothness information contained therein. While this novel approach leads to substantial savings in communication, it is limited to sparsification as it crucially depends on the linearity of the compression operator. In this work, we generalize their smoothness-aware compression strategy to arbitrary unbiased compression operators, which also includes sparsification. Specializing our results to stochastic quantization, we guarantee significant savings in communication complexity compared to standard quantization. In particular, we prove that block quantization with n blocks theoretically outperforms single block quantization, leading to a reduction in communication complexity by an 𝒪(n) factor, where n is the number of nodes in the distributed system. Finally, we provide extensive numerical evidence with convex optimization problems that our smoothness-aware quantization strategies outperform existing quantization schemes as well as the aforementioned smoothness-aware sparsification strategies with respect to all relevant success measures: the number of iterations, the total amount of bits communicated, and wall-clock time.To address the high communication costs of distributed machine learning, a large body of work has been devoted in recent years to designing various compression strategies, such as sparsification and quantization, and optimization algorithms capable of using them. Recently, Safaryan et al. [2021] pioneered a dramatically different compression design approach: they first use the local training data to form local smoothness matrices and then propose to design a compressor capable of exploiting the…
Organizer
NeurIPS 2022
Account · 953 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Teaching Algorithmic Reasoning via In-context Learning
Watch later
Hattie Zhou, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Data augmentation for efficient learning from parametric experts
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Skill, or Style? Classification of Fetal Sonography Eye-Tracking Data
Watch later
Clare Teng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Deconstructing Distributions: A Pointwise Framework of Learning
Watch later
Gal Kaplun, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Manifold Interpolating Optimal-Transport Flows for Trajectory Inference
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Approximation with CNNs in Sobolev Space: with Applications to Classification
Watch later
Guohao Shen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%