[Paper Review] First Analysis of Local GD on Heterogeneous Data

Ahmed Khaled, Konstantin Mishchenko+1 more|arXiv (Cornell University)|Sep 10, 2019

Stochastic Gradient Optimization Techniques19 references67 citations

TL;DR

The paper provides the first convergence analysis of local gradient descent for minimizing the average of L-smooth convex functions with heterogeneous data across devices, showing that in a low accuracy regime it matches the communication complexity of standard gradient descent.

ABSTRACT

We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions. Problems of this form and local gradient descent as a solution method are of importance in federated learning, where each function is based on private data stored by a user on a mobile device, and the data of different users can be arbitrarily heterogeneous. We show that in a low accuracy regime, the method has the same communication complexity as gradient descent.

Motivation & Objective

• Motivate and analyze local gradient descent for the average objective f(x)= (1/M) sum_m f_m(x) where each f_m is convex and L-smooth.
• Address non-i.i.d. data across devices in federated learning and understand convergence without bounded gradient assumptions.
• Derive a neighborhood convergence bound and relate communication complexity to accuracy and data heterogeneity.
• Highlight the role of the heterogeneity measure sigma^2 = (1/M) sum_m ||∇f_m(x_*)||^2 in governing convergence.
• Provide insights into how local steps (H) and synchronization affect convergence compared to centralized GD.

Proposed method

• Define the local gradient descent algorithm with periodic synchronization across M devices and local updates between synchronizations.
• Prove a recursion on the optimality gap using r_t = x̂_t - x_* and bounds on V_t (iterate variance) and g_t (average gradient).
• Introduce the key bound f(x̄_T) - f(x_*) ≤ 2||x_0 - x_*||^2 /(γT) + 24 γ^2 σ^2 H^2 L for appropriate γ and synchronization interval H.
• Derive the optimal communication complexity bound by minimizing T/H under constraints γ ≤ 1/(4LH) and γ-related conditions.
• Relate local GD to minibatch SGD through Corollary 1, showing 1/√(MT) and 1/√T rates under certain choices of H.

Experimental results

Research questions

• RQ1 Can local gradient descent converge for minimizing the average of heterogeneous L-smooth convex functions without bounded gradient assumptions?
• RQ2 How does data heterogeneity, captured by σ^2, influence convergence and communication complexity of Local GD?
• RQ3 What is the relationship between the number of local steps H, the synchronization interval, and the overall communication rounds needed to reach a target accuracy?
• RQ4 How does Local GD compare to standard GD and minibatch SGD in terms of convergence rate and communication efficiency under non-i.i.d. data?

Key findings

• The method attains a convergence bound f(x̄_T) - f(x_*) ≤ 2||x_0 - x_*||^2 /(γT) + 24 γ^2 σ^2 H^2 L for a suitable γ.
• When ε is not too small (ε ≥ 3σ^2/L), the communication complexity matches that of gradient descent up to constants.
• If ε < 3σ^2/L, the communication complexity scales as O(√L σ / ε^{3/2}).
• Corollary shows a 1/√(MT) rate in terms of communication rounds with appropriate H and γ, illustrating a trade-off with minibatch SGD.
• Local GD behaves like minibatch SGD with a neighborhood size determined by σ^2, emphasizing the impact of heterogeneity on convergence.
• Empirical results on LIBSVM datasets show the theory matches practice, with local methods advantageous when precise accuracy is not required under higher communication costs.