Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
Apr 4, 2021
Speakers
Kiwan Maeng
Speaker · 0 followers
Shivam Bharuka
Speaker · 0 followers
Isabel Gao
Speaker · 0 followers
Mark Jeffrey
Speaker · 0 followers
Vikram Saraph
Speaker · 0 followers
Bor-Yiing Su
Speaker · 0 followers
Caroline Trippel
Speaker · 0 followers
Jiyan Yang
Speaker · 0 followers
Mike Rabbat
Speaker · 0 followers
Brandon Lucia
Speaker · 0 followers
Carole-Jean Wu
Speaker · 0 followers
About
The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node fails during training, improving failure-related overheads. The paper is the first to the extent of our knowledge to perform a data-driven, in-depth analysis of applying partial recovery to recommendation models and identified a trade-off between accuracy and performance. Motivated by the analysis, we present CPR, a partial recovery training system that can reduce the training time and maintain the desired level of model accuracy by (1) estimating the benefit of partial recovery, (2) selecting an appropriate checkpoint saving interval, and (3) prioritizing to save updates of more frequently accessed parameters. Two variants of CPR, CPR-MFU and CPR-SSU, reduce the checkpoint-related overhead from 8.2--8.5% to 0.53--0.68% compared to full recovery, on a setup emulating the failure pattern and overhead of a production-scale cluster. While reducing overhead significantly, CPR achieves model quality on par with the more expensive full recovery scheme, training the state-of-the-art recommendation model using Criteo’s Ads CTR dataset. Our preliminary results also suggest that CPR can speed up training on a real production-scale cluster, without notably degrading the accuracy.The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node fails during training, improving failure-related overheads. The paper is the first to the extent of our knowledge to perform a data-driven, in-depth analysis of applying partial recovery to recommendation models and identified a trade-off between accuracy and performance. Motiva…
Organizer
MLSys 2021
Account · 159 followers
About MLSys 2021
The Conference on Machine Learning and Systems targets research at the intersection of machine learning and systems. The conference aims to elicit new connections amongst these fields, including identifying best practices and design principles for learning systems, as well as developing novel learning methods and theory tailored to practical machine learning workflows.
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
Oral: Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models
Watch later
Shang Wang, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Oral: Value Learning for Throughput Optimization of Deep Learning Workloads
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Larq Compute Engine: Design, Benchmark and Deploy State-of-the-Art Binarized Neural Networks
Watch later
Tom Bannink, …
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Al and The Virtuous Cycle of Compute
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Graphcore’s IPU and GNNs
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Applying Circuit IR Compilers and Tools (CIRCT) to ML Applications
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%