[Paper Review] On Efficient Constructions of Checkpoints

Yu Chen, Zhenming Liu+2 more|arXiv (Cornell University)|Sep 28, 2020

Formal Methods in Verification4 citations

TL;DR

This paper proposes LC-Checkpoint, a lossy compression scheme for deep learning model checkpoints that simultaneously maximizes compression rate and optimizes recovery speed under SGD training. By combining exponent-based quantization, priority promotion, and Huffman coding on gradient deltas, it achieves up to 28× compression and 5.77× faster recovery than SCAR, enabling efficient, frequent checkpointing with minimal overhead.

ABSTRACT

Efficient construction of checkpoints/snapshots is a critical tool for training and diagnosing deep learning models. In this paper, we propose a lossy compression scheme for checkpoint constructions (called LC-Checkpoint). LC-Checkpoint simultaneously maximizes the compression rate and optimizes the recovery speed, under the assumption that SGD is used to train the model. LC-Checkpointuses quantization and priority promotion to store the most crucial information for SGD to recover, and then uses a Huffman coding to leverage the non-uniform distribution of the gradient scales. Our extensive experiments show that LC-Checkpoint achieves a compression rate up to $28 imes$ and recovery speedup up to $5.77 imes$ over a state-of-the-art algorithm (SCAR).

Motivation & Objective

• To address the growing I/O and storage burden caused by frequent model checkpointing in large-scale deep learning training.
• To design a lossy compression scheme that maximizes compression rate while ensuring fast recovery for SGD-based training.
• To leverage statistical patterns in gradient updates and learning dynamics to identify and preserve only the most critical information for recovery.
• To minimize the runtime overhead of checkpointing by integrating techniques from distributed training and model compression.
• To enable fine-grained, high-frequency checkpointing that reduces rework after failures or crashes.

Proposed method

• The method uses delta-encoding to store only the difference between consecutive model states, reducing redundancy.
• It applies exponent-based quantization to compress parameter updates while preserving critical training information.
• A priority promotion mechanism identifies and retains parameters with significant updates, discarding those with negligible changes.
• The remaining data is encoded using Huffman coding to exploit non-uniform gradient scale distributions and achieve near-information-theoretic compression.
• The scheme is designed to be efficient and scalable, incurring less than 4% overhead per iteration in experiments.
• It assumes SGD as the training algorithm and focuses on minimizing recovery time post-failure, not reconstruction accuracy.

Experimental results

Research questions

• RQ1 Can we design a lossy compression scheme for model checkpoints that achieves high compression while enabling fast recovery during SGD training?
• RQ2 How can we identify and preserve only the most critical information in model parameter updates for training recovery?
• RQ3 To what extent can techniques from distributed training and model compression be combined to optimize checkpointing efficiency?
• RQ4 What is the trade-off between compression rate, recovery speed, and runtime overhead in frequent checkpointing?
• RQ5 Can we achieve near-optimal compression by leveraging the non-uniform distribution of gradient scales in deep learning?

Key findings

• LC-Checkpoint achieves a maximum compression rate of 28× compared to full model checkpoints.
• It delivers up to 5.77× faster recovery speed than the state-of-the-art SCAR algorithm.
• In the MovieLens25M experiment, LC-Checkpoint saved 546 seconds of training time compared to full checkpointing, with only 4 seconds of additional overhead per iteration.
• The method reduces checkpoint size to below 10% of the original size using 3-bit priority promotion.
• The approach incurs negligible runtime overhead—under 4% per iteration—making it suitable for frequent checkpointing.
• The scheme outperforms SCAR by eliminating redundant information through selective compression based on training dynamics.