[Paper Review] CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery

Kiwan Maeng, Shivam Bharuka+9 more|arXiv (Cornell University)|Nov 5, 2020

Privacy-Preserving Technologies in Data74 references4 citations

TL;DR

CPR is a partial recovery training system for deep learning recommendation models that reduces checkpoint-related overhead by allowing non-failed nodes to continue training without reloading checkpoints, achieving up to 8.5% overhead reduction and maintaining model accuracy comparable to full recovery. It optimizes checkpoint intervals and prioritizes frequently accessed parameters to balance performance and accuracy.

ABSTRACT

The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node fails during training, improving failure-related overheads. The paper is the first to the extent of our knowledge to perform a data-driven, in-depth analysis of applying partial recovery to recommendation models and identified a trade-off between accuracy and performance. Motivated by the analysis, we present CPR, a partial recovery training system that can reduce the training time and maintain the desired level of model accuracy by (1) estimating the benefit of partial recovery, (2) selecting an appropriate checkpoint saving interval, and (3) prioritizing to save updates of more frequently accessed parameters. Two variants of CPR, CPR-MFU and CPR-SSU, reduce the checkpoint-related overhead from 8.2-8.5% to 0.53-0.68% compared to full recovery, on a configuration emulating the failure pattern and overhead of a production-scale cluster. While reducing overhead significantly, CPR achieves model quality on par with the more expensive full recovery scheme, training the state-of-the-art recommendation model using Criteo's Ads CTR dataset. Our preliminary results also suggest that CPR can speed up training on a real production-scale cluster, without notably degrading the accuracy.

Motivation & Objective

• Address the high computational overhead caused by full checkpoint recovery in large-scale distributed recommendation training.
• Reduce failure-related training delays and resource consumption in production clusters with frequent node failures.
• Maintain model accuracy while minimizing checkpoint overhead through intelligent partial recovery mechanisms.
• Identify and optimize the trade-off between training performance and model quality in partial recovery systems.
• Design a scalable, production-ready system that outperforms full recovery in both efficiency and fault tolerance.

Proposed method

• Propose CPR, a partial recovery system that allows non-failed nodes to proceed without reloading checkpoints after a failure.
• Estimate the benefit of partial recovery using a user-specified target model quality (PLS) to guide checkpoint interval selection.
• Prioritize saving updates of more frequently accessed parameters to reduce checkpoint size and I/O overhead.
• Implement two variants—CPR-MFU and CPR-SSU—using different optimization strategies for checkpoint saving and recovery.
• Leverage the iterative-convergent nature of deep learning training to tolerate temporary inconsistencies introduced by partial recovery.
• Use data-driven analysis of production failure patterns to guide system design and validate performance under realistic conditions.

Experimental results

Research questions

• RQ1 What is the trade-off between training performance and model accuracy when applying partial recovery in recommendation model training?
• RQ2 How can checkpoint saving intervals be optimized to minimize overhead while maintaining target model quality?
• RQ3 Which parameter access patterns most significantly impact checkpoint overhead and accuracy degradation in partial recovery?
• RQ4 Can partial recovery reduce checkpoint-related overhead without degrading model quality compared to full recovery?
• RQ5 How does CPR scale with increasing numbers of training nodes under realistic failure distributions?

Key findings

• Checkpoint-related overhead in full recovery consumes an average of 12% of total training time, with up to 43% slowdown for the worst 5% of jobs.
• CPR reduces checkpoint-related overhead from 8.2–8.5% (full recovery) to just 0.53–0.68% on production-scale cluster configurations.
• CPR maintains model accuracy on par with full recovery when training the state-of-the-art recommendation model on the Criteo CTR dataset.
• CPR demonstrates better scalability than full recovery: overhead decreases with more nodes due to reduced relative update loss.
• The system achieves significant efficiency gains—1,156 machine-years of computation were spent on failure handling in 17,000 jobs over 30 days, which CPR can significantly reduce.
• Preliminary results suggest CPR can speed up training in real production clusters without notable accuracy degradation.