[Paper Review] DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression

Hanlin Tang, Xiangru Lian+3 more|arXiv (Cornell University)|May 15, 2019

Stochastic Gradient Optimization Techniques94 citations

TL;DR

DoubleSqueeze analyzes and proves convergence for parallel error-compensated SGD with two-pass (worker and parameter server) compression, showing linear speedup and enhanced tolerance to compression bias/noise.

ABSTRACT

A standard approach in large scale machine learning is distributed stochastic gradient training, which requires the computation of aggregated stochastic gradients over multiple nodes on a network. Communication is a major bottleneck in such applications, and in recent years, compressed stochastic gradient methods such as QSGD (quantized SGD) and sparse SGD have been proposed to reduce communication. It was also shown that error compensation can be combined with compression to achieve better convergence in a scheme that each node compresses its local stochastic gradient and broadcast the result to all other nodes over the network in a single pass. However, such a single pass broadcast approach is not realistic in many practical implementations. For example, under the popular parameter server model for distributed learning, the worker nodes need to send the compressed local gradients to the parameter server, which performs the aggregation. The parameter server has to compress the aggregated stochastic gradient again before sending it back to the worker nodes. In this work, we provide a detailed analysis on this two-pass communication model and its asynchronous parallel variant, with error-compensated compression both on the worker nodes and on the parameter server. We show that the error-compensated stochastic gradient algorithm admits three very nice properties: 1) it is compatible with an \emph{arbitrary} compression technique; 2) it admits an improved convergence rate than the non error-compensated stochastic gradient methods such as QSGD and sparse SGD; 3) it admits linear speedup with respect to the number of workers. The empirical study is also conducted to validate our theoretical results.

Motivation & Objective

• Motivate reducing communication bottlenecks in distributed stochastic gradient training.
• Extend error-compensation to both workers and parameter servers in a two-pass communication model.
• Prove convergence and linear speedup for the proposed DoubleSqueeze algorithm under non-convex loss.
• Provide empirical validation that supports theoretical convergence and practical bandwidth savings.

Proposed method

• Introduce DoubleSqueeze where both workers and the parameter server perform error-compensated compression on communicated gradients.
• Use a compression operator Q_ω[·] that may be biased or unbiased, with error vectors δ^{(i)} and δ at workers and server to compensate information loss.
• Show the global update can be written as x_{t+1}=x_t-γ∇f(x_t)+γξ_t-γΩ_{t-1}+γΩ_t, where Ω_t and ξ_t capture compression errors and stochastic gradient variance.
• Prove that with appropriate assumptions (Lipschitz gradient, bounded variance, bounded compression error), DoubleSqueeze achieves a convergence rate with linear speedup in the number of workers n.
• Corollary provides a rate of O(σ/√(nT)) plus terms depending on ε and T, indicating faster convergence under parallelism and tolerance to compression errors.

Experimental results

Research questions

• RQ1 Can error compensation be effectively extended to both workers and the parameter server in a two-pass compression setting?
• RQ2 Does the parallel, double-pass error-compensated SGD achieve linear speedup with respect to the number of workers?
• RQ3 How does DoubleSqueeze compare to non-error-compensated SGD and other compressed SGD methods under non-convex loss?
• RQ4 What compression operators (biased or unbiased) can be used within DoubleSqueeze while preserving convergence?
• RQ5 What empirical bandwidth savings and convergence behavior arise in practice for common models and datasets?

Key findings

• DoubleSqueeze achieves convergence with linear speedup proportional to the number of workers n.
• The method tolerates compression bias and noise better than non-error-compensated SGD, improving convergence in compressed settings.
• There are only n rounds of communication per iteration with full compression on both sides, enabling significant bandwidth savings.
• Theoretical results extend to non-convex loss, matching SGD-like rates in the presence of compression.
• Empirical results on CIFAR-10 with ResNet-18 show similar convergence to uncompressed SGD while achieving faster per-iteration time under limited bandwidth, outperforming non-compensated methods under bandwidth constraints.
• When using 1-bit and top-k compression, DoubleSqueeze maintains competitive training and testing performance with substantial speedups under restricted network conditions.