[Paper Review] Variance Reduced Local SGD with Lower Communication Complexity

Xianfeng Liang, Shuheng Shen+4 more|arXiv (Cornell University)|Dec 30, 2019

Advanced Image and Video Retrieval Techniques37 references89 citations

TL;DR

VRL-SGD reduces communication in distributed non-identical-data training by incorporating variance reduction into Local SGD, achieving a lower communication complexity and linear iteration speedup.

ABSTRACT

To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data distribution on workers is non-identical, Local SGD requires $O(T^{\frac{3}{4}} N^{\frac{3}{4}})$ communications to maintain its \emph{linear iteration speedup} property, where $T$ is the total number of iterations and $N$ is the number of workers. In this paper, we propose Variance Reduced Local SGD (VRL-SGD) to further reduce the communication complexity. Benefiting from eliminating the dependency on the gradient variance among workers, we theoretically prove that VRL-SGD achieves a \emph{linear iteration speedup} with a lower communication complexity $O(T^{\frac{1}{2}} N^{\frac{3}{2}})$ even if workers access non-identical datasets. We conduct experiments on three machine learning tasks, and the experimental results demonstrate that VRL-SGD performs impressively better than Local SGD when the data among workers are quite diverse.

Motivation & Objective

• Motivate accelerating distributed SGD under non-identical data distributions to reduce communication.
• Develop a Local SGD variant that mitigates gradient variance across workers without extra assumptions.
• Establish theoretical convergence guarantees and linear speedup with reduced communication.
• Demonstrate practical efficacy on standard ML tasks with non-identical data distributions.

Proposed method

• Introduce VRL-SGD, a Local SGD variant with a variance-reduction component to align local and global gradients.
• Compute a gradient correction term Delta_i to approximate global gradient differences across communication periods.
• Update local models with a stochastic gradient corrected by Delta_i to reduce inter-worker variance.
• Allow local updates for k steps between communications to lower communication rounds.
• Provide theoretical convergence analysis showing O(T^{-1/2}N^{-1/2}) rate and improved communication complexity.
• Show that VRL-SGD reduces communication complexity from O(T^{3/4}N^{3/4}) to O(T^{1/2}N^{3/2}) in non-identical data scenarios.

Experimental results

Research questions

• RQ1 Can variance reduction eliminate the dependency on gradient variance among workers in Local SGD under non-identical data?
• RQ2 What is the resulting communication complexity and iteration speedup of VRL-SGD compared to Local SGD and S-SGD?
• RQ3 Do VRL-SGD guarantees extend to non-convex objectives and identical data scenarios?
• RQ4 How does VRL-SGD perform empirically on non-identical data tasks (image, text, transfer learning) versus baselines?

Key findings

• VRL-SGD achieves linear iteration speedup with lower communication complexity O(T^{1/2}N^{3/2}) for non-identical data.
• It does not require the bounded-gradient-variance or identical-data assumptions used by prior Local SGD analyses.
• Empirical results on MNIST, DBPedia, and tiny ImageNet show VRL-SGD outperforms Local SGD when data are non-identical and matches S-SGD/Local SGD when data are identical.
• Theoretical results show convergence rate O(1/√(NT)) for non-convex objectives with appropriate learning rate and communication period settings.
• A warm-up variant (VRL-SGD-W) can reduce the dependence on non-iid initialization (C term) and tighten convergence.