[Paper Review] QSGD: Randomized Quantization for Communication-Optimal Stochastic Gradient Descent

Dan Alistarh, Jerry Li+2 more|arXiv (Cornell University)|Oct 7, 2016

Stochastic Gradient Optimization Techniques15 references68 citations

TL;DR

QSGD is a communication-efficient stochastic gradient descent method that uses randomized quantization to compress gradient updates, enabling provably convergent training with sublinear communication cost in model dimension. It achieves asymptotically optimal communication efficiency while maintaining competitive performance on real-world deep learning tasks.

ABSTRACT

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to excellent scalability properties of this algorithm, and to its efficiency in the context of training deep neural networks. A fundamental barrier for parallelizing large-scale SGD is the fact that the cost of communicating the gradient updates between nodes can be very large. Consequently, lossy compresion heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always provably converge, and it is not clear whether they are optimal. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes which allow the compression of gradient updates at each node, while guaranteeing convergence under standard assumptions. QSGD allows the user to trade off compression and convergence time: it can communicate a sublinear number of bits per iteration in the model dimension, and can achieve asymptotically optimal communication cost. We complement our theoretical results with empirical data, showing that QSGD can significantly reduce communication cost, while being competitive with standard uncompressed techniques on a variety of real tasks.

Motivation & Objective

• To address the high communication cost in distributed SGD by enabling efficient gradient compression.
• To design a compression scheme that guarantees convergence under standard assumptions, unlike many heuristic quantization methods.
• To enable a tunable trade-off between communication cost and convergence speed through adjustable quantization levels.
• To achieve asymptotically optimal communication complexity in terms of bits per iteration relative to model dimension.
• To empirically validate that quantized gradients maintain performance comparable to full-precision SGD on real deep learning tasks.

Proposed method

• QSGD uses randomized quantization to compress gradient vectors into a low-bit representation before transmission between nodes.
• It maps each gradient component to a quantized value using a probabilistic rounding scheme that preserves the gradient's expected value.
• The method allows users to control the number of bits per gradient component, enabling a trade-off between compression and accuracy loss.
• The quantization scheme is designed such that the expected value of the quantized gradient equals the original gradient, ensuring unbiased updates.
• Communication cost scales sublinearly with model dimension, achieving asymptotic optimality in terms of bits per iteration.
• The algorithm maintains convergence guarantees under standard stochastic gradient descent assumptions, including Lipschitz continuity and bounded variance.

Experimental results

Research questions

• RQ1 Can gradient quantization be designed to guarantee convergence in distributed SGD, unlike many heuristic approaches?
• RQ2 What is the theoretical minimum communication cost required for stochastic gradient descent, and can it be achieved?
• RQ3 How does the trade-off between quantization bit-rate and convergence speed affect training performance?
• RQ4 Can quantized gradients maintain competitive accuracy and convergence speed on real deep learning tasks?
• RQ5 What is the optimal quantization strategy that balances communication efficiency and convergence stability?

Key findings

• QSGD achieves asymptotically optimal communication cost, scaling sublinearly with model dimension in terms of bits per iteration.
• The method guarantees convergence under standard assumptions, providing theoretical safety where heuristic compression methods often fail.
• Empirical results show QSGD maintains training accuracy competitive with full-precision SGD on various real-world deep learning tasks.
• By adjusting the number of bits per gradient component, QSGD enables a tunable trade-off between communication cost and convergence speed.
• The randomized quantization scheme ensures unbiased gradient estimates, preserving convergence properties despite compression.
• QSGD significantly reduces communication overhead without sacrificing model performance, making it suitable for large-scale distributed training.