GBA: A Tuning-free Approach to Switch between Synchronous and Asynchronous Training for Recommendation Models
Nov 28, 2022
Speakers
Yuanxing Zhang
Řečník · 0 sledujících
Yufeng Cai
Řečník · 0 sledujících
Pengjie Wang
Řečník · 0 sledujících
Hongbo Deng
Řečník · 0 sledujících
About
High-concurrency asynchronous training upon parameter server (PS) architecture and high-performance synchronous training upon all-reduce (AR) architecture are the most commonly deployed distributed training modes for recommender systems. Although the synchronous AR training is designed to have higher training efficiency, the asynchronous PS training would be a better choice on training speed when there are stragglers (slow workers) in the shared cluster, especially under limited computing resources. To take full advantages of these two training modes, an ideal way is to switch between them upon the cluster status. We find two obstacles to a tuning-free approach: the different distribution of the gradient values and the stale gradients from the stragglers. In this paper, we propose Global Batch gradients Aggregation (GBA) over PS, which aggregates and applies gradients with the same global batch size as the synchronous training. A token-control process is implemented to assemble the gradients and decay the gradients with severe staleness. We provide the convergence analysis to reveal that GBA has comparable convergence properties with the synchronous training, and demonstrate the robustness of GBA the recommendation models against the gradient staleness. Experiments on three industrial-scale recommendation tasks show that GBA is an effective tuning-free approach for switching. Compared to the state-of-the-art derived asynchronous training, GBA achieves up to 0.2High-concurrency asynchronous training upon parameter server (PS) architecture and high-performance synchronous training upon all-reduce (AR) architecture are the most commonly deployed distributed training modes for recommender systems. Although the synchronous AR training is designed to have higher training efficiency, the asynchronous PS training would be a better choice on training speed when there are stragglers (slow workers) in the shared cluster, especially under limited computing resour…
Organizer
NeurIPS 2022
Účet · 962 sledujících
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Exact learning dynamics of deep linear networks with prior knowledge
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Distributed Learning of Conditional Quantiles in RKHS
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Manifold learning via landmark diffusion and clinical applications
Zhlédnout později
Yu-Ting Lin, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Locally Constrained Representations in Reinforcement Learning
Zhlédnout později
Somjit Nath, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Sparse Hypergraph Community Detection Thresholds in Stochastic Block Model
Zhlédnout později
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %
Deconfounded Imitation Learning
Zhlédnout později
Risto Vuorio, …
Pro uložení prezentace do věčného trezoru hlasovalo 0 diváků, což je 0.0 %