[Paper Review] Distributed Learning with Compressed Gradient Differences

Konstantin Mishchenko, Eduard Gorbunov+2 more|arXiv (Cornell University)|Jan 26, 2019

Stochastic Gradient Optimization Techniques18 references108 citations

TL;DR

Introduces the DIANA algorithm, which compresses gradient differences for distributed learning, with theoretical convergence guarantees in strongly convex and nonconvex settings and empirical comparisons to QSGD and TernGrad.

ABSTRACT

Training large machine learning models requires a distributed computing approach, with communication of the model updates being the bottleneck. For this reason, several methods based on the compression (e.g., sparsification and/or quantization) of updates were recently proposed, including QSGD (Alistarh et al., 2017), TernGrad (Wen et al., 2017), SignSGD (Bernstein et al., 2018), and DQGD (Khirirat et al., 2018). However, none of these methods are able to learn the gradients, which renders them incapable of converging to the true optimum in the batch mode. In this work we propose a new distributed learning method -- DIANA -- which resolves this issue via compression of gradient differences. We perform a theoretical analysis in the strongly convex and nonconvex settings and show that our rates are superior to existing rates. We also provide theory to support non-smooth regularizers study the difference between quantization schemes. Our analysis of block-quantization and differences between $\ell_2$ and $\ell_{\infty}$ quantization closes the gaps in theory and practice. Finally, by applying our analysis technique to TernGrad, we establish the first convergence rate for this method.

Motivation & Objective

• Address the communication bottleneck in distributed optimization for large-scale models.
• Develop a compression-based method that preserves convergence to the true optimum.
• Provide theoretical guarantees in both strongly convex and nonconvex settings.
• Handle non-smooth regularizers and analyze block quantization versus dimension-based quantization.
• Situate the method relative to existing compression schemes like QSGD, TernGrad, and DQGD.

Proposed method

• Propose DIANA, which compresses gradient differences rather than full gradients to reduce communication.
• Introduce memory vectors h_i^k at each worker to learn the gradient at the optimum and ensure differences converge to zero.
• Use unbiased stochastic gradients g_i^k with bounded variance and form deltas Δ_i^k = g_i^k − h_i^k.
• Apply p-quantization to Δ_i^k to obtain compressed updates sent to the server.
• Aggregate quantized updates to form a global direction and perform a proximal gradient step with momentum (v^k).
• Provide a framework that includes block quantization and analyzes the impact of different p-norms (p ≥ 1) for quantization.

Experimental results

Research questions

• RQ1 Can gradient-difference compression ensure convergence to the true optimum in distributed settings with heterogeneous data?
• RQ2 What are the convergence rates for DIANA in strongly convex and nonconvex regimes under smoothness and regularization assumptions?
• RQ3 How do different quantization schemes (p-quantization, block quantization) affect communication and convergence?
• RQ4 How does DIANA compare to existing compressed-update methods like QSGD and TernGrad in theory and experiments?
• RQ5 What is the impact of non-smooth regularizers and momentum on convergence and practicality?

Key findings

• DIANA achieves convergence rates that can be competitive with or superior to existing compression-based methods under both strongly convex and nonconvex objectives.
• The gradient-difference compression with memory learning of the optimum gradient enables learning to the true optimum despite compression.
• Block quantization and higher p-norms (e.g., p = ∞) can reduce communication without sacrificing convergence, under appropriate parameter choices.
• The analysis provides new convergence guarantees for Terngrad (TernGrad) and 1-bit QSGD under broader assumptions than previously shown.
• Empirical experiments show DIANA with momentum typically outperforms QSGD, TernGrad, and DQGD in logistic regression and CIFAR-10 settings.