[Paper Review] Faster Convergence of Local SGD for Over-Parameterized Models

Tiancheng Qin, S. Rasoul Etesami+1 more|arXiv (Cornell University)|Jan 30, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper establishes faster convergence rates for Local SGD in over-parameterized models under heterogeneous data, proving exponential convergence for strongly convex losses ($\mathcal{O}(\exp(-T))$) and $\mathcal{O}(1/T)$ for general convex losses under mild data similarity, significantly improving prior bounds of $\mathcal{O}(\exp(-T/K))$ and no known rates for general/non-convex cases.

ABSTRACT

Modern machine learning architectures are often highly expressive. They are usually over-parameterized and can interpolate the data by driving the empirical loss close to zero. We analyze the convergence of Local SGD (or FedAvg) for such over-parameterized models in the heterogeneous data setting and improve upon the existing literature by establishing the following convergence rates. For general convex loss functions, we establish an error bound of $Ø(1/T)$ under a mild data similarity assumption and an error bound of $Ø(K/T)$ otherwise, where $K$ is the number of local steps and $T$ is the total number of iterations. For non-convex loss functions we prove an error bound of $Ø(K/T)$. These bounds improve upon the best previous bound of $Ø(1/\sqrt{nT})$ in both cases, where $n$ is the number of nodes, when no assumption on the model being over-parameterized is made. We complete our results by providing problem instances in which our established convergence rates are tight to a constant factor with a reasonably small stepsize. Finally, we validate our theoretical results by performing large-scale numerical experiments that reveal the convergence behavior of Local SGD for practical over-parameterized deep learning models, in which the $Ø(1/T)$ convergence rate of Local SGD is clearly shown.

Motivation & Objective

• To address the gap in theoretical convergence guarantees for Local SGD in over-parameterized models under heterogeneous data.
• To explain the faster empirical convergence of Local SGD compared to Minibatch SGD in large-scale deep learning.
• To establish tighter convergence rates under the interpolation assumption common in modern over-parameterized models.
• To validate theoretical findings with large-scale numerical experiments on real and synthetic data.
• To demonstrate tightness of the derived bounds via problem instances under small stepsize schemes.

Proposed method

• Analyzes Local SGD under the interpolation assumption, where empirical loss can be driven to zero.
• Introduces a novel analysis framework combining consensus error $V_t$ and gradient variation $h_t$ to bound convergence.
• Uses a telescoping sum argument on the error $e_t = f(x^t) - f(x^*)$ to derive cumulative bounds.
• Applies a stepsize $\eta = \frac{1}{3KL\rho}$ to balance gradient and consensus error terms.
• Leverages the smoothness and bounded gradient assumptions to control error propagation across local steps.
• Establishes bounds on $V_t$ using recursive unrolling and the property $\rho \geq 1$ to simplify the final error expression.

Experimental results

Research questions

• RQ1 Can Local SGD achieve faster convergence than Minibatch SGD in over-parameterized models under heterogeneous data?
• RQ2 What is the tightest possible convergence rate for Local SGD in the strongly convex, general convex, and non-convex settings under over-parameterization?
• RQ3 How does data heterogeneity affect convergence, and can mild data similarity assumptions improve rates?
• RQ4 Are the derived convergence rates tight, and can they be matched by problem instances?
• RQ5 Can the theoretical bounds be validated empirically in large-scale deep learning settings?

Key findings

• For strongly convex loss functions, Local SGD achieves an exponential convergence rate of $\mathcal{O}(\exp(-T))$, improving upon the prior $\mathcal{O}(\exp(-T/K))$.
• For general convex losses, an $\mathcal{O}(1/T)$ rate is established under a mild data similarity assumption, and $\mathcal{O}(K/T)$ otherwise.
• For non-convex losses, the paper proves an $\mathcal{O}(K/T)$ convergence rate, which was previously unknown in the over-parameterized setting.
• The theoretical bounds are shown to be tight to within a constant factor via constructed problem instances under small stepsize schemes.
• Large-scale numerical experiments on real and synthetic data validate the theoretical convergence behavior of Local SGD in practical over-parameterized deep learning models.
• The analysis reveals that the client drift issue in heterogeneous settings can be mitigated under over-parameterization, enabling faster convergence.