[Paper Review] Gradient Sparsification for Communication-Efficient Distributed Optimization

Jianqiao Wangni, Jialei Wang+2 more|arXiv (Cornell University)|Oct 26, 2017

Stochastic Gradient Optimization TechniquesComputer Science28 references207 citations

TL;DR

The paper introduces a convex formulation to sparsify stochastic gradients for distributed optimization, balancing sparsity and variance, with efficient algorithms and empirical validation on convex and non-convex problems.

ABSTRACT

Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost we propose a convex optimization formulation to minimize the coding length of stochastic gradients. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for approximate solution, with theoretical guaranteed for sparseness. Experiments on $\ell_2$ regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

Motivation & Objective

• Reduce communication cost in distributed stochastic optimization by sparsifying gradients while preserving unbiasedness.
• Formulate a convex problem to optimize sparsity under a variance budget.
• Develop fast algorithms with sparsity guarantees to solve the optimal sparsification problem.
• Demonstrate effectiveness on convex problems (logistic regression, SVM) and deep neural networks.
• Provide theoretical insights on sparsity and variance trade-offs.

Proposed method

• sparsify a gradient g by keeping coordinate i with probability pi and amplifying kept coordinates by gi/pi to keep an unbiased Q(g).
• Formulate a linear program to minimize total sparsity sum pi subject to a variance constraint sum gi^2/pi ≤ (1+ε) sum gi^2.
• Derive a closed-form optimal solution where pi = min(λ|gi|, 1) and show a dominant set Sk of top-magnitude coordinates are kept with pi=1.
• Provide two practical algorithms (Algorithm 2 closed-form; Algorithm 3 greedy) to compute the probability vector p efficiently.
• Discuss a coding strategy that splits coordinates into Sk and non-Sk for efficient transmission, including representation of amplified and non-amplified parts.
• Outline the integration into synchronous (Algorithm 1) and asynchronous distributed optimization settings.
• Offer theoretical sparsity guarantees under (ρ, s)-approximately sparse gradients and corresponding coding-length bounds.

Experimental results

Research questions

• RQ1 Can stochastic gradient sparsification achieve substantial communication reductions without significantly harming convergence?
• RQ2 What is the optimal trade-off between gradient sparsity and variance under a given variance budget?
• RQ3 How can one compute near-optimal sparsification probabilities in linear time with guarantees?
• RQ4 How does gradient sparsification interact with standard distributed optimization methods (SGD, SVRG, ADAM) and coding strategies?
• RQ5 Do the proposed sparsification methods generalize to convex and non-convex problems and real-world datasets?

Key findings

• The sparsification method yields an unbiased Q(g) by sampling coordinates with probabilities p_i and scaling by 1/p_i for kept coordinates.
• A convex formulation minimizes sparsity under a variance budget, yielding p_i = min(λ|g_i|, 1) with a data-dependent λ.
• Two practical algorithms (a closed-form solution and a greedy algorithm) compute p efficiently with sparsity guarantees.
• The approach reduces coding length and communication while maintaining competitive convergence on ℓ2-regularized logistic regression, SVMs, and CNNs.
• Empirical results show the method outperforms uniform sampling and is competitive with QSGD across varying sparsity levels and data distributions; stronger sparsity leads to larger gains in communication efficiency.
• Theoretical analysis links (ρ, s)-approximately sparse gradients to an upper bound on expected sparsity and a controlled variance increase.