[Paper Review] Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication

Anastasiia Koloskova, Sebastian U. Stich+1 more|arXiv (Cornell University)|Feb 1, 2019

Stochastic Gradient Optimization TechniquesComputer Science185 citations

TL;DR

Introduces Choco-SGD and Choco-Gossip, decentralized SGD and gossip algorithms with compressed communication, proving convergence rates and linear consensus under biased/unbiased compression and various network topologies.

ABSTRACT

We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over $n$ machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communication bottleneck, the nodes compress (e.g. quantize or sparsify) their model updates. We cover both unbiased and biased compression operators with quality denoted by $ω\leq 1$ ($ω=1$ meaning no compression). We (i) propose a novel gossip-based stochastic gradient descent algorithm, CHOCO-SGD, that converges at rate $\mathcal{O}\left(1/(nT) + 1/(T δ^2 ω)^2 ight)$ for strongly convex objectives, where $T$ denotes the number of iterations and $δ$ the eigengap of the connectivity matrix. Despite compression quality and network connectivity affecting the higher order terms, the first term in the rate, $\mathcal{O}(1/(nT))$, is the same as for the centralized baseline with exact communication. We (ii) present a novel gossip algorithm, CHOCO-GOSSIP, for the average consensus problem that converges in time $\mathcal{O}(1/(δ^2ω) \log (1/ε))$ for accuracy $ε> 0$. This is (up to our knowledge) the first gossip algorithm that supports arbitrary compressed messages for $ω> 0$ and still exhibits linear convergence. We (iii) show in experiments that both of our algorithms do outperform the respective state-of-the-art baselines and CHOCO-SGD can reduce communication by at least two orders of magnitudes.

Motivation & Objective

• Investigate decentralized stochastic optimization where data is distributed across multiple nodes connected by a fixed graph.
• Develop communication-efficient algorithms that use compressed updates to mitigate bandwidth bottlenecks.
• Establish convergence guarantees that match centralized mini-batch SGD in key terms while leveraging network topology and compression quality.

Proposed method

• Propose Choco-SGD, a gossip-based decentralized SGD with compressed communication, achieving rate O(1/(nT) + 1/(T δ^2 ω)^2) for strongly convex objectives.
• Introduce Choco-Gossip, a gossip algorithm for average consensus with arbitrary compression, proving linear convergence rate O(1/(δ^2 ω) log(1/ε)).
• Allow both unbiased and biased compression operators characterized by ω ≤ 1, where ω = 1 indicates no compression.
• Provide a unified convergence analysis showing the first decentralized stochastic rate matching centralized mini-batch SGD in leading term.
• Demonstrate that network topology (via δ) and compression (via ω) affect only higher-order terms in the convergence rate.
• Validate theoretically and experimentally that Choco-SGD and Choco-Gossip outperform state-of-the-art baselines and substantially reduce communication.

Experimental results

Research questions

• RQ1 Can decentralized stochastic optimization achieve the same leading-order convergence speed as centralized mini-batch SGD despite compression and network topology?
• RQ2 How do compression quality (ω) and network connectivity (δ) influence convergence rates in decentralized SGD and gossip algorithms?
• RQ3 Is it possible to design gossip and SGD schemes that support biased compression while preserving convergence to the true solution?
• RQ4 Do practical implementations of Choco-SGD and Choco-Gossip provide empirical advantages over existing decentralized methods under communication constraints?

Key findings

• Choco-SGD converges at rate O(1/(nT) + 1/(T δ^2 ω)^2) for strongly convex objectives, matching the leading term of centralized mini-batch SGD.
• Choco-Gossip converges linearly with rate O(1/(δ^2 ω) log(1/ε)) for average consensus under compressed communication.
• The first term O(1/(nT)) in Choco-SGD is preserved regardless of compression quality and network connectivity, mirroring centralized performance.
• Choco-SGD and Choco-Gossip outperform state-of-the-art baselines (ECD-SGD, DCD-SGD) in experiments, with Choco-SGD achieving substantial communication reductions.
• Choco-Gossip supports arbitrary compressed messages for ω>0 and achieves linear convergence, unlike prior methods requiring near-perfect quantization.