[Paper Review] On the Utility of Gradient Compression in Distributed Training Systems

Saurabh Agarwal, Hongyi Wang+2 more|arXiv (Cornell University)|Feb 28, 2021

Stochastic Gradient Optimization Techniques94 references19 citations

TL;DR

This paper evaluates gradient compression in distributed training and finds that, despite significant system optimizations like overlapping computation and communication, only 6 out of 200+ configurations show speedup over optimized synchronous SGD. The key insight is that high encode-decode overhead in compression methods limits their benefit, especially at high bandwidths, and future gains depend on faster compute or reduced compression latency.

ABSTRACT

A rich body of prior work has highlighted the existence of communication bottlenecks in synchronous data-parallel training. To alleviate these bottlenecks, a long line of recent work proposes gradient and model compression methods. In this work, we evaluate the efficacy of gradient compression methods and compare their scalability with optimized implementations of synchronous data-parallel SGD across more than 200 different setups. Surprisingly, we observe that only in 6 cases out of more than 200, gradient compression methods provide speedup over optimized synchronous data-parallel training in the typical data-center setting. We conduct an extensive investigation to identify the root causes of this phenomenon, and offer a performance model that can be used to identify the benefits of gradient compression for a variety of system setups. Based on our analysis, we propose a list of desirable properties that gradient compression methods should satisfy, in order for them to provide a meaningful end-to-end speedup.

Motivation & Objective

• To assess the real-world utility of gradient compression in modern distributed training systems.
• To compare gradient compression methods against highly optimized synchronous SGD implementations with overlapping computation and communication.
• To identify why gradient compression often fails to deliver speedups despite theoretical communication savings.
• To develop a performance model that predicts when gradient compression will provide end-to-end speedup.
• To define desirable properties for future gradient compression methods to achieve meaningful scalability.

Proposed method

• Empirically evaluated three gradient compression methods—sign SGD, MSTop-K, and PowerSGD—across 200+ configurations using ResNet-50, ResNet-101, and BERT.
• Used PyTorch v1.8's communication hook interface to enable overlapping of gradient compression with backward pass computation.
• Measured end-to-end iteration times across varying batch sizes, network bandwidths (1–30 Gbps), and GPU counts (up to 96).
• Built an analytical performance model that accounts for computation, communication, and encode-decode overhead to predict compression utility.
• Simulated effects of improved compute (up to 4× faster) and reduced encode-decode time with varying compression ratios.
• Used NCCL for efficient all-reduce communication and benchmarked against PyTorch DDP with native optimizations.

Experimental results

Research questions

• RQ1 Under what system conditions does gradient compression provide end-to-end speedup over optimized synchronous SGD?
• RQ2 Why do existing gradient compression methods fail to deliver speedups despite reducing communication volume?
• RQ3 How do system-level optimizations like computation-communication overlap affect the viability of gradient compression?
• RQ4 What is the impact of network bandwidth and compute speed on the effectiveness of gradient compression?
• RQ5 What trade-offs between compression ratio and encode-decode time are most beneficial for performance?

Key findings

• Only 6 out of 200+ experimental configurations showed speedup from gradient compression over optimized synchronous SGD at typical data-center bandwidths.
• Encode-decode time for compression methods (e.g., 50+ ms) exceeds the available window to hide communication, limiting speedup potential.
• PowerSGD outperforms sign SGD in scalability because it is compatible with all-reduce, while sign SGD’s lack of all-reduce support leads to higher latency.
• At network bandwidths above 9 Gbps, PowerSGD becomes slower than synchronous SGD due to its fixed encode-decode overhead.
• With 4× faster compute, PowerSGD with Rank-4 can achieve up to 1.75× speedup, showing that future hardware advances may unlock compression benefits.
• Reducing encode-decode time—even at the cost of lower compression ratios—significantly improves performance, indicating that latency reduction is more critical than compression ratio.