Oral: Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
Apr 4, 2021
Speakers
Shang Wang
Speaker · 0 followers
Peiming Yang
Speaker · 0 followers
Yuxuan Zheng
Speaker · 0 followers
Gennady Pekhimenko
Speaker · 0 followers
About
Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years. To reduce this training cost and optimize the cluster-wide hardware resource usage, we collect and analyze “real-world” GPU cluster usage statistics. Our study reveals that single-accelerator training jobs can dominate the cluster-wide resource consumption when launched repetitively (e.g., for hyper-parameter tuning) while severely under-utilizing the hardware. This is because DL researchers and practitioners often lack the required expertise to independently optimize their own workloads. Fortunately, we observe that such workloads have the following unique characteristics: (i) the models among jobs often have the same types of operators with the same shapes, and (ii) the inter-model horizontal fusion of such operators is mathematically equivalent to other already well-optimized operators. Thus, to help DL researchers and practitioners effectively and easily improve the hardware utilization of their novel DL training workloads, we propose Horizontally Fused Training Array (HFTA). HFTA is a new DL framework extension library that horizontally fuses the models from different repetitive jobs deeply down to operators, and then trains those models simultaneously on a shared accelerator. On three emerging DL training workloads and state-of-the-art accelerators (GPUs and TPUs), HFTA demonstrates strong effectiveness on squeezing out hardware utilization and achieves up to 15.1x higher training throughput vs. the standard practice of running each job on a separate accelerator.Driven by the tremendous effort in researching novel deep learning (DL) algorithms, the training cost of developing new models increases staggeringly in recent years. To reduce this training cost and optimize the cluster-wide hardware resource usage, we collect and analyze “real-world” GPU cluster usage statistics. Our study reveals that single-accelerator training jobs can dominate the cluster-wide resource consumption when launched repetitively (e.g., for hyper-parameter tuning) while severely…
Organizer
MLSys 2021
Account · 159 followers
Categories
AI & Data Science
Category · 10.8k presentations
About MLSys 2021
The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Oral: SUOD: Accelerating Large-scale Unsupervised Heterogeneous Outlier Detection
Watch later
Yue Zhao, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Oral: CODE: Compiler-based Neuron-aware Ensemble training
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
A Deep Learning Based Cost Model for Automatic Code Optimization
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
MERCI: Efficient Embedding Reduction on Commodity Hardware via Sub-Query Memoization
Watch later
Yejin Lee, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Oral: IOS: Inter-Operator Scheduler for CNN Acceleration
Watch later
Yaoyao Ding, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Trustworthy AI
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%