[Paper Review] Communication Compression for Decentralized Training

Hanlin Tang, Shaoduo Gan+3 more|arXiv (Cornell University)|Mar 17, 2018

Stochastic Gradient Optimization TechniquesComputer Science185 citations

TL;DR

The paper introduces two quantized decentralized SGD algorithms (DCD-PSGD and ECD-PSGD) that compress exchanged models while preserving convergence, achieving rate O(1/√(nT)) and showing significant speedups under high latency and low bandwidth in experiments on CIFAR-10 with ResNet-20.

ABSTRACT

Optimizing distributed learning systems is an art of balancing between computation and communication. There have been two lines of research that try to deal with slower networks: {\em communication compression} for low bandwidth networks, and {\em decentralization} for high latency networks. In this paper, We explore a natural question: {\em can the combination of both techniques lead to a system that is robust to both bandwidth and latency?} Although the system implication of such combination is trivial, the underlying theoretical principle and algorithm design is challenging: unlike centralized algorithms, simply compressing exchanged information, even in an unbiased stochastic way, within the decentralized network would accumulate the error and fail to converge. In this paper, we develop a framework of compressed, decentralized training and propose two different strategies, which we call {\em extrapolation compression} and {\em difference compression}. We analyze both algorithms and prove both converge at the rate of $O(1/\sqrt{nT})$ where $n$ is the number of workers and $T$ is the number of iterations, matching the convergence rate for full precision, centralized training. We validate our algorithms and find that our proposed algorithm outperforms the best of merely decentralized and merely quantized algorithm significantly for networks with {\em both} high latency and low bandwidth.

Motivation & Objective

• Motivate robust distributed training that combines decentralization with communication compression to handle both high latency and low bandwidth networks.
• Develop two compressed decentralized SGD algorithms (DCD-PSGD and ECD-PSGD) that maintain convergence guarantees.
• Provide theoretical convergence analysis showing rate matching centralized training under certain conditions.
• Empirically validate that the proposed methods outperform purely decentralized or purely quantized approaches in challenging networks.

Proposed method

• Formulate decentralized optimization with n nodes and Lipschitz-gradient objectives.
• Introduce two quantized decentralized SGD algorithms: DCD-PSGD (difference compression) and ECD-PSGD (extrapolation compression).
• Impose assumptions: symmetric double stochastic communication matrix W with spectral gap ρ, Lipschitz gradients, bounded gradient variance σ² and ζ², and unbiased stochastic compression with signal-to-noise parameter α.
• For DCD-PSGD, compress the difference z_t^(i) = x_t^(i+1/2) − x_t^(i) and update replicas of neighbors’ models accordingly, ensuring convergence via Theorem 1 and Corollaries.
• For ECD-PSGD, use extrapolated z-values to transmit neighbor estimates and prove convergence under bounded compression noise Assumption 2, achieving comparable rates to DCD-PSGD with enhanced robustness to aggressive compression.
• Derive convergence rates: leading term O(σ/√(nT)) plus terms involving ζ, α, ρ, and γ; provide corollaries giving O(1/√(nT)) overall rate and linear speedup with the number of nodes.

Experimental results

Research questions

• RQ1 Can the combination of decentralized training and unbiased compression achieve convergence without error accumulation?
• RQ2 What convergence rates can be established for compressed decentralized SGD, and how do they compare to centralized and uncompressed decentralized baselines?
• RQ3 How do the two proposed strategies (difference compression and extrapolation compression) differ in robustness and performance under varying network conditions?
• RQ4 Do the proposed methods exhibit linear speedup with the number of workers in practical settings?

Key findings

• Two compressed decentralized SGD algorithms (DCD-PSGD and ECD-PSGD) converge with rate roughly O(1/√(nT)).
• ECD-PSGD is more robust to aggressive compression while DCD-PSGD can have better rates when data variation across nodes is large; however, too aggressive compression may cause divergence in DCD-PSGD.
• Leading convergence term matches centralized parallel SGD, indicating linear speedup with the number of nodes.
• Theoretical results are complemented by experiments showing that decentralized, low-precision training can outperform Allreduce in high-latency or low-bandwidth networks.