[Paper Review] Local SGD Converges Fast and Communicates Little

Sebastian U. Stich|arXiv (Cornell University)|May 24, 2018

Stochastic Gradient Optimization Techniques53 references222 citations

TL;DR

This paper proves that Local SGD achieves linear speedup in the number of workers and mini-batch size, while reducing communication rounds by up to a factor of O(sqrt(T/(Kb))) compared to mini-batch SGD, for convex problems.

ABSTRACT

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis. We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speedup in the number of workers and mini-batch size. The number of communication rounds can be reduced up to a factor of T^{1/2}---where T denotes the number of total steps---compared to mini-batch SGD. This also holds for asynchronous implementations. Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.

Motivation & Objective

• Motivate and address the communication bottleneck in large-scale distributed SGD.
• Analyze local SGD where workers evolve independently and synchronize periodically.
• prove that averaging local updates yields linear speedups in computation and reduces communication rounds.
• Provide practical guidelines for step sizes and synchronization frequency.

Proposed method

• Model the problem as finite-sum convex optimization with L-smooth and mu-strongly convex f.
• Run K parallel SGD sequences with mini-batch size b and synchronize by averaging every H iterations.
• Define a virtual averaged sequence to analyze fluctuations and show variance reduction from averaging.
• Derive convergence rate for the averaged iterate under appropriate stepsizes, showing O(1/(KT B)) behavior asymptotically.
• Extend to mini-batch local SGD by replacing sigma^2 with sigma^2/b to account for variance reduction.
• Show an asynchronous variant where synchronization is not required every step and provide analogous convergence guarantees.

Experimental results

Research questions

• RQ1 Does averaging locally evolved SGD sequences yield a linear speedup relative to running SGD on a single worker?
• RQ2 How does the communication interval H affect convergence, and can we quantify the trade-off between computation and communication?
• RQ3 Can Local SGD achieve the same convergence rate as mini-batch SGD while reducing communication rounds, and how do mini-batch size b and number of workers K interact?
• RQ4 Do asynchronous local SGD schemes retain similar convergence properties under delayed or non-synchronous updates?

Key findings

• Local SGD achieves a linear speedup in the number of workers K and the local mini-batch size b for convex problems.
• With a synchronization interval H = O(sqrt(T/(Kb))), the method reduces the number of communication rounds by up to a factor of O(sqrt(T/(Kb))) without sacrificing asymptotic convergence.
• The convergence rate of the averaged iterate is E[f(x̂_T)] - f* = O((G^2)/(mu b K T)) plus lower-order terms, matching parallel mini-batch SGD in computation while reducing communication.
• For mini-batch local SGD, the variance term scales as sigma^2/b, preserving the linear speedup with respect to b and K (rate ~ O(1/(KTb))).
• An asynchronous version shows similar rates under bounded delays, implying speedups without strict synchronization.
• The theoretical results provide practical guidelines on choosing H and stepsizes to balance compute and communication.