Oral: Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
Apr 4, 2021
Speakers
Shang Wang
Speaker · 0 followers
Peiming Yang
Speaker · 0 followers
Yuxuan Zheng
Speaker · 0 followers
Gennady Pekhimenko
Speaker · 0 followers
About
Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years. To reduce this training cost and optimize the cluster-wide hardware resource usage, we collect and analyze “real-world” GPU cluster usage statistics. Our study reveals that single-accelerator training jobs can dominate the cluster-wide resource consumption when launched repetitively (e.g., for hyper-parameter tuning) while severely under-utilizing the hardware. This is because DL researchers and practitioners often lack the required expertise to independently optimize their own workloads. Fortunately, we observe that such workloads have the following unique characteristics: (i) the models among jobs often have the same types of operators with the same shapes, and (ii) the inter-model horizontal fusion of such operators is mathematically equivalent to other already well-optimized operators. Thus, to help DL researchers and practitioners effectively and easily improve the hardware utilization of their novel DL training workloads, we propose Horizontally Fused Training Array (HFTA). HFTA is a new DL framework extension library that horizontally fuses the models from different repetitive jobs deeply down to operators, and then trains those models simultaneously on a shared accelerator. On three emerging DL training workloads and state-of-the-art accelerators (GPUs and TPUs), HFTA demonstrates strong effectiveness on squeezing out hardware utilization and achieves up to 15.1x higher training throughput vs. the standard practice of running each job on a separate accelerator.Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years. To reduce this training cost and optimize the cluster-wide hardware resource usage, we collect and analyze “real-world” GPU cluster usage statistics. Our study reveals that single-accelerator training jobs can dominate the cluster-wide resource consumption when launched repetitively (e.g., for hyper-parameter tuning) while severely…
Organizer
MLSys 2021
Account · 159 followers
Categories
AI & Data Science
Category · 10.8k presentations
About MLSys 2021
The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Session 6: Benchmarks, Cost models, and Profiling
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Graph Representation Learning for Chip Design
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Designing and Optimizing AI Systems for Deep Learning Recommendation and Beyond
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Panel Session
Watch later
Tom St John, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
Watch later
Kiwan Maeng, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Challenges with Running DNN Workloads with Hardware Simulators
Watch later
David Kaeli, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%