[Paper Review] Minibatch vs Local SGD for Heterogeneous Distributed Learning

Blake Woodworth, Kumar Kshitij Patel+1 more|arXiv (Cornell University)|Jun 8, 2020

Stochastic Gradient Optimization Techniques24 references45 citations

TL;DR

The paper analyzes Local SGD and Minibatch SGD under heterogeneous distributed data and limited communication, showing Minibatch SGD generally dominates Local SGD, with accelerated Minibatch SGD optimal for high heterogeneity and Local SGD potentially improving only in near-homogeneous settings.

ABSTRACT

We analyze Local SGD (aka parallel or federated SGD) and Minibatch SGD in the heterogeneous distributed setting, where each machine has access to stochastic gradient estimates for a different, machine-specific, convex objective; the goal is to optimize w.r.t. the average objective; and machines can only communicate intermittently. We argue that, (i) Minibatch SGD (even without acceleration) dominates all existing analysis of Local SGD in this setting, (ii) accelerated Minibatch SGD is optimal when the heterogeneity is high, and (iii) present the first upper bound for Local SGD that improves over Minibatch SGD in a non-homogeneous regime.

Motivation & Objective

• Motivate distributed training where data are split across heterogeneous machines with limited communication.
• Compare convergence guarantees of Minibatch SGD versus Local SGD in heterogeneous settings.
• Characterize regimes where Local SGD can or cannot outperform Minibatch SGD.
• Provide refined analysis showing when acceleration helps and when it does not.
• Identify regimes where new methods may be needed for heterogeneity.”

Proposed method

• Define the heterogeneous distributed optimization setup with M machines and local objectives F_m and the average objective F.
• Analyze unbiasedness and variance of Minibatch SGD gradients and derive convergence bounds that are independent of heterogeneity metric zeta_*.
• Provide accelerated Minibatch SGD bounds and show they match or improve standard Minibatch SGD under convex/strongly convex assumptions.
• Present existing Local SGD analyses in the heterogeneous setting and derive lower bounds showing limitations.
• Introduce a refined heterogeneity measure bar(zeta)^2 to recover improvement of Local SGD in near-homogeneous regimes.
• Show a dual-stepsize framework (inner/outer stepsizes) that interpolates between Minibatch and Local SGD and yields improved guarantees when optimally chosen.”

Experimental results

Research questions

• RQ1 Does Local SGD outperform Minibatch SGD in heterogeneous distributed learning with limited communication?
• RQ2 How does heterogeneity (zeta_* and bar(zeta)) affect the relative performance of Local SGD and Minibatch SGD?
• RQ3 Is accelerated Minibatch SGD optimal in highly heterogeneous settings?
• RQ4 Under what conditions can Local SGD improve over Minibatch SGD, if at all, in near-homogeneous regimes?
• RQ5 Can interpolation via inner/outer stepsizes bridge the gap between the two methods?
• RQ6 What are fundamental lower bounds for distributed zero-respecting algorithms in this setting?

Key findings

• Minibatch SGD and Accelerated Minibatch SGD achieve convergence bounds that are independent of the heterogeneity measure zeta_* and thus are immune to data heterogeneity.
• In the heterogeneous setting, existing Local-SGD analyses cannot beat Minibatch SGD except in very near-homogeneous regimes; for sufficiently large heterogeneity, Local SGD is strictly worse.
• A refined heterogeneity measure bar(zeta)^2 enables Local SGD to outperform Minibatch SGD in near-homogeneous scenarios, representing the first such result.
• Accelerated Minibatch SGD is optimal for moderately high heterogeneity, as any Local SGD improvement is precluded by lower bounds in many regimes.
• A distributed zero-respecting algorithm lower bound shows Accelerated Minibatch SGD is optimal for large heterogeneity, with potential improvements only when bar(zeta)^2 is small; a clear gap remains for intermediate heterogeneity levels.