[Paper Review] Optimal Gradient Compression for Distributed and Federated Learning

Alyazeed Albasyoni, Mher Safaryan+2 more|arXiv (Cornell University)|Oct 7, 2020

Sparse and Compressive Sensing TechniquesEngineering40 references36 citations

TL;DR

This paper analyzes the fundamental rate-distortion trade-off for gradient compression in distributed and federated learning, and introduces efficient compressors (Sparse Dithering and Spherical Compression) that approach the lower bounds in worst- and average-case analyses.

ABSTRACT

Communicating information, like gradient vectors, between computing nodes in distributed and federated learning is typically an unavoidable burden, resulting in scalability issues. Indeed, communication might be slow and costly. Recent advances in communication-efficient training algorithms have reduced this bottleneck by using compression techniques, in the form of sparsification, quantization, or low-rank approximation. Since compression is a lossy, or inexact, process, the iteration complexity is typically worsened; but the total communication complexity can improve significantly, possibly leading to large computation time savings. In this paper, we investigate the fundamental trade-off between the number of bits needed to encode compressed vectors and the compression error. We perform both worst-case and average-case analysis, providing tight lower bounds. In the worst-case analysis, we introduce an efficient compression operator, Sparse Dithering, which is very close to the lower bound. In the average-case analysis, we design a simple compression operator, Spherical Compression, which naturally achieves the lower bound. Thus, our new compression schemes significantly outperform the state of the art. We conduct numerical experiments to illustrate this improvement.

Motivation & Objective

• Motivate and formalize the rate-distortion trade-off in gradient compression for distributed/federated learning.
• Characterize worst-case and average-case lower bounds on bits per vector given distortion.
• Propose efficient compression operators that are near-optimal and practical in high dimensions.
• Analyze the impact of compression on iterative convergence and total communication.
• Provide experimental validation showing communication-convergence trade-offs.

Proposed method

• Define compression operators as encoder–decoder pairs and categorize them into unbiased, contractive, and strictly contractive classes.
• Derive worst-case lower bounds on bits required for a given distortion using the uncertainty principle for communication compression.
• Construct Sparse Dithering as a near-optimal, efficient compressor and analyze both deterministic (biased) and randomized (unbiased) variants.
• Introduce Spherical Compression as an average-case optimal compressor achieving the lower bound within small overhead.
• Compare with existing methods through theoretical bounds and empirical experiments on distributed optimization tasks.

Experimental results

Research questions

• RQ1 What is the fundamental lower bound relating distortion and bits for gradient compression in high dimensions?
• RQ2 Can we design compression operators that nearly achieve these bounds while remaining computationally efficient?
• RQ3 How do compression parameters (distortion level, bit budget) affect iteration complexity and total communication in distributed learning?
• RQ4 Are there practical compressors that are optimal in the average-case sense and outperform existing approaches?
• RQ5 What are the trade-offs between biased/unbiased compressors in terms of convergence and communication efficiency?

Key findings

• A tight asymptotic lower bound shows b* (alpha, d) ≈ -log P(alpha, d) + log d + (1/2) log log d + e, with e small, establishing near-optimality in worst-case settings.
• Sparse Dithering (SD) achieves near-optimal worst-case performance with efficient encoding and, in biased and randomized variants, achieves significant bandwidth reductions.
• Deterministic SD with nu = 1/10 communicates at most 30 + log d + 3.35 d bits and is within 1.69 d bits of theoretical optimum.
• Randomized SD (unbiased) achieves at most 30 + log d + (log 3 + 1/(2√ω)) d bits in expectation, enabling substantial bandwidth savings (≈9.9× in some settings).
• Spherical Compression (SC) attains the average-case lower bound within less than 3 extra bits, delivering near-optimal average communication.
• Experiments demonstrate meaningful reductions in communication while maintaining convergence behavior across ridge and logistic regression tasks.