[Paper Review] Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Yujun Lin, Song Han+3 more|arXiv (Cornell University)|Dec 5, 2017

Advanced Neural Network Applications37 references645 citations

TL;DR

Deep Gradient Compression reduces gradient communication by 270× to 600× using momentum correction, local gradient clipping, momentum factor masking, and warm-up training, while preserving accuracy across CNNs and RNNs.

ABSTRACT

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile. Code is available at: https://github.com/synxlin/deep-gradient-compression.

Motivation & Objective

• Motivate the need to reduce communication bandwidth in synchronous distributed SGD for large-scale training.
• Propose a gradient compression approach that preserves accuracy under high sparsity.
• Introduce mechanisms to mitigate convergence and staleness issues caused by sparse updates.

Proposed method

• Gradient sparsification to transmit only large gradients while accumulating small ones locally.
• Encode sparse gradients using 32-bit nonzeros and 16-bit run lengths of zeros.
• Momentum correction to align sparse updates with dense momentum SGD updates.
• Local gradient clipping to bound explosion risks within each node.
• Momentum factor masking to reduce stale momentum impact from delayed gradients.
• Warm-up training to gradually increase sparsity and stabilize early training.

Experimental results

Research questions

• RQ1 Can gradient exchange be reduced dramatically (up to orders of magnitude) without accuracy loss in diverse tasks?
• RQ2 How can sparsity-induced convergence issues be mitigated in distributed SGD with momentum?
• RQ3 What combination of techniques yields the best trade-off between bandwidth reduction and model performance across CNNs and RNNs?
• RQ4 What runtime strategies (e.g., hierarchical thresholding) enable scalable sparse gradient selection?

Key findings

• Gradient compression ratios from 270× to 600× are achieved without accuracy loss across tasks and datasets.
• On ImageNet with ResNet-50, compression reaches 277× with negligible accuracy drop compared to baseline (Top-1: 58.17% baseline vs 58.20% with DGC; Top-5: 80.19% baseline vs 80.20% with DGC).
• For ResNet-110 on CIFAR-10 with 4 GPUs, baseline Top-1 accuracy is 93.75% and DGC achieves 93.87% ( +0.12%).
• On ImageNet with 4 GPUs and 256 total batch size, baseline Top-1 is 92.92% and DGC is 93.28% (+0.37%).
• Language modeling on Penn Treebank shows perplexity 72.30 (baseline) vs 72.24 (DGC) with gradient size 0.42 MB (462× compression).
• Speech recognition on LibriSpeech shows WER 9.45% (test-clean) with baseline vs 9.06% (test-clean) with DGC, and gradient size 0.74 MB (608× compression).