[Paper Review] QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

Dan Alistarh, Demjan Grubic+3 more|arXiv (Cornell University)|Oct 7, 2016

Stochastic Gradient Optimization Techniques907 citations

TL;DR

QSGD introduces stochastic gradient quantization with Elias coding to reduce communication in data-parallel SGD, providing convergence guarantees and practical speedups for deep networks without sacrificing accuracy.

ABSTRACT

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to excellent scalability properties of this algorithm, and to its efficiency in the context of training deep neural networks. A fundamental barrier for parallelizing large-scale SGD is the fact that the cost of communicating the gradient updates between nodes can be very large. Consequently, lossy compression heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always provably converge, and it is not clear whether they are optimal. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes which allow the compression of gradient updates at each node, while guaranteeing convergence under standard assumptions. QSGD allows the user to trade off compression and convergence time: it can communicate a sublinear number of bits per iteration in the model dimension, and can achieve asymptotically optimal communication cost. We complement our theoretical results with empirical data, showing that QSGD can significantly reduce communication cost, while being competitive with standard uncompressed techniques on a variety of real tasks. In particular, experiments show that gradient quantization applied to training of deep neural networks for image classification and automated speech recognition can lead to significant reductions in communication cost, and end-to-end training time. For instance, on 16 GPUs, we are able to train a ResNet-152 network on ImageNet 1.8x faster to full accuracy. Of note, we show that there exist generic parameter settings under which all known network architectures preserve or slightly improve their full accuracy when using quantization.

Motivation & Objective

• Motivate scalable data-parallel SGD by addressing communication bottlenecks in gradient exchange.
• Develop a quantized SGD framework with convergence guarantees under convex and non-convex objectives.
• Provide a practical encoding scheme to compress quantized gradients efficiently without breaking convergence.
• Demonstrate applicability to deep neural networks and show substantial end-to-end training time reductions.

Proposed method

• Propose stochastic gradient quantization Q_s(v) with s levels to preserve unbiasedness and control variance.
• Encode quantized gradients using a lossless Elias-based coding scheme exploiting the distributional properties of quantized values.
• Allow bucketed quantization to control variance via bucket size d and scale by vector norms for stability.
• Derive theoretical bounds: variance inflation by min(n/s^2, sqrt(n)/s) and communication length per round.
• Extend to variants including QSVRG (variance-reduced) and non-convex settings with convergence guarantees.
• Provide practical implementation notes for bucketing, max-normalization, and GPU-friendly coding.

Experimental results

Research questions

• RQ1 How does gradient quantization affect convergence guarantees in parallel SGD under convex and non-convex objectives?
• RQ2 What is the trade-off between communication bits per iteration and convergence/variance in QSGD?
• RQ3 Can stochastic quantization with efficient coding achieve substantial communication reductions with preserved accuracy on deep networks?
• RQ4 How do QSGD variants (including variance-reduced versions) perform in practice compared to full-precision SGD?

Key findings

• QSGD achieves significant communication reductions with convergence guarantees, enabling practical speedups on GPUs.
• In the dense regime (s = sqrt(n)), the per-iteration communication can be as low as 2.8n + 32 bits with at most a 2x variance increase.
• Two extremes: either O(sqrt(n) (log n + O(1))) expected bits per iteration with at most O(sqrt(n)) variance increase, or ≤ 2.8n + 32 bits per iteration with about double the iterations.
• Empirical results show substantial training time reductions across ImageNet classifiers and LSTM speech models, with minimal or no loss in accuracy (e.g., AlexNet on 16 GPUs: 4x less communication and 2.5x faster epoch; ResNet-152 on 16 GPUs: ≈2x faster end-to-end).
• QSGD variants like QSVRG retain exponential convergence properties and offer favorable per-epoch communication relative to problem conditioning.
• Quantization noise can even slightly improve accuracy in some settings, consistent with observed benefits of gradient noise in deep learning.