PipeMare: Asynchronous Pipeline Parallel DNN Training
Apr 4, 2021
Speakers
Bowen Yang
Speaker · 0 followers
Jian Zhang
Speaker · 1 follower
Jonathan Li
Speaker · 0 followers
Christopher Ré
Speaker · 5 followers
Christopher Aberger
Speaker · 0 followers
Christopher De Sa
Speaker · 1 follower
About
Pipeline parallelism when training neural networks enables models to be partitioned spatially, which can lead to overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing pipeline parallel training techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices will be necessary on the emerging class of new dataflow hardware accelerators. We devise PipeMare, a simple yet robust training method that tolerates asynchronous updates during pipeline parallel execution without sacrificing utilization or memory, which allows efficient use of fine-grained pipeline parallelism. Concretely, when tested on ResNet and Transformer networks, asynchrony enables PipeMare to use up to 2.7x less memory or get 14.3x higher pipeline utilization, with similar model quality, when compared to state-of-the-art synchronous pipeline parallel training techniques.Pipeline parallelism when training neural networks enables models to be partitioned spatially, which can lead to overall higher hardware utilization. Unfortunately, to preserve the statistical efficiency of sequential training, existing pipeline parallel training techniques sacrifice hardware efficiency by decreasing pipeline utilization or incurring extra memory costs. In this paper, we investigate to what extent these sacrifices will be necessary on the emerging class of new dataflow hardware…
Organizer
MLSys 2021
Account · 159 followers
About MLSys 2021
The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
Watch later
Haichen Shen, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
sensAI: ConvNets Decomposition via Class Parallelism for Fast Inference on Live Data
Watch later
Guanhua Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Learned Performance Model for a Deep Learning Accelerator
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
CODE: Compiler-based Neuron-aware Ensemble training
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Session 10: Techniques, and more Techniques
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Welcome to JOURNE
Watch later
Udit Gupta, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%