[论文解读] Check-N-Run: A Checkpointing System for Training Recommendation Models.

Assaf Eisenman, Kiran Kumar Matam+6 more|arXiv (Cornell University)|Oct 17, 2020

Privacy-Preserving Technologies in Data参考文献 18被引用 6

一句话总结

Check-N-Run 是一种用于训练大规模推荐模型的可扩展检查点系统，通过结合增量检查点和模型量化技术，将写入带宽减少6–17倍，存储容量减少2.5–8倍，显著提升了检查点效率并降低了总体拥有成本，同时不损失训练精度。

ABSTRACT

Checkpoints play an important role in training recommendation systems at scale. They are important for many use cases, including failure recovery to ensure rapid training progress, and online training to improve inference prediction accuracy. Checkpoints are typically written to remote, persistent storage. Given the typically large and ever-increasing recommendation model sizes, the checkpoint frequency and effectiveness is often bottlenecked by the storage write bandwidth and capacity, as well as the network bandwidth. We present Check-N-Run, a scalable checkpointing system for training large recommendation models. Check-N-Run uses two primary approaches to address these challenges. First, it applies incremental checkpointing, which tracks and checkpoints the modified part of the model. On top of that, it leverages quantization techniques to significantly reduce the checkpoint size, without degrading training accuracy. These techniques allow Check-N-Run to reduce the required write bandwidth by 6-17x and the required capacity by 2.5-8x on real-world models at Facebook, and thereby significantly improve checkpoint capabilities while reducing the total cost of ownership.

研究动机与目标

• 解决大规模推荐模型训练中检查点技术面临的可扩展性挑战。
• 降低因频繁且大规模的检查点写入导致的高存储和网络带宽开销。
• 在最小化检查点大小和写入频率的同时保持训练精度。
• 通过优化检查点 I/O 降低训练系统的总体拥有成本。
• 通过高效的检查点机制实现更快的故障恢复和更有效的在线训练。

提出的方法

• 采用增量检查点技术，仅追踪并持久化模型中已修改的部分，从而减少冗余写入。
• 应用量化技术对模型权重进行压缩，显著减少检查点大小。
• 系统将增量检查点与训练后量化相结合，以保持模型精度。
• 通过优先处理已更改的参数并利用低精度表示，优化检查点写入。
• 架构设计旨在与现有训练框架无缝集成，仅需极少修改。
• 系统根据模型更新模式动态平衡检查点频率与大小。

实验结果

研究问题

• RQ1 如何在大规模推荐模型训练中减少检查点开销？
• RQ2 增量检查点在多大程度上能够减少写入带宽和存储容量？
• RQ3 量化技术是否能在减少检查点大小的同时保持训练精度？
• RQ4 结合增量检查点与量化技术对系统效率和成本有何影响？
• RQ5 该系统在真实工作负载条件下大规模运行时表现如何？

主要发现

• 在 Facebook 的真实推荐模型上，Check-N-Run 将所需写入带宽减少了6–17倍。
• 系统将所需存储容量减少了2.5–8倍，显著降低了基础设施成本。
• 尽管采用了激进的量化和增量检查点技术，模型精度仍得以保持。
• 增量检查点与量化技术的结合使得更频繁且高效的检查点成为可能。
• 系统提升了故障恢复速度，并支持更有效的在线训练。
• 该方案在保持训练稳定性和性能的同时实现了显著的成本节约。