[Paper Review] On the Convergence of Local Descent Methods in Federated Learning

Farzin Haddadpour, Mehrdad Mahdavi|arXiv (Cornell University)|Oct 31, 2019

Stochastic Gradient Optimization Techniques38 references169 citations

TL;DR

The paper analyzes the convergence of local GD/SGD with periodic averaging in federated learning under heterogeneous data, proving convergence rates and identifying how gradient diversity bounds enable variance reduction and linear speedups. It covers centralized and networked settings, including non-convex and PL-conditioned objectives.

ABSTRACT

In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., heterogeneous or non i.i.d. data samples). In this paper, we generalize the local stochastic and full gradient descent with periodic averaging-- originally designed for homogeneous distributed optimization, to solve nonconvex optimization problems in federated learning. Although scant research is available on the effectiveness of local SGD in reducing the number of communication rounds in homogeneous setting, its convergence and communication complexity in heterogeneous setting is mostly demonstrated empirically and lacks through theoretical understating. To bridge this gap, we demonstrate that by properly analyzing the effect of unbiased gradients and sampling schema in federated setting, under mild assumptions, the implicit variance reduction feature of local distributed methods generalize to heterogeneous data shards and exhibits the best known convergence rates of homogeneous setting both in general nonconvex and under {\pl}~ condition (generalization of strong-convexity). Our theoretical results complement the recent empirical studies that demonstrate the applicability of local GD/SGD to federated learning. We also specialize the proposed local method for networked distributed optimization. To the best of our knowledge, the obtained convergence rates are the sharpest known to date on the convergence of local decant methods with periodic averaging for solving nonconvex federated optimization in both centralized and networked distributed optimization.

Motivation & Objective

• Motivate the study of communication-efficient federated optimization with heterogeneous data distributions.
• Generalize local GD/SGD with periodic averaging to nonconvex objectives in federated settings.
• Establish convergence rates under bounded gradient diversity and PL condition.
• Specialize results to centralized, decentralized (networked), and sampled-device federated configurations.

Proposed method

• Propose Local Federated Descent (LFD) with periodic averaging, parameterized by E (local updates), K (sampled devices), and q (device weights).
• Specialize LFD to Local Federated GD (LFGD) and Local Federated SGD (LFSGD) with both full and stochastic gradient settings.
• Introduce Weighted Gradient Diversity Λ(w,q) to quantify heterogeneity and derive conditions on learning rate and E for convergence.
• Derive convergence guarantees for non-convex objectives and for PL-conditioned non-convex objectives.
• Extend analysis to networked distributed optimization where devices communicate with direct neighbors.

Experimental results

Research questions

• RQ1 How does heterogeneity across local data shards affect convergence of local descent with periodic averaging in federated learning?
• RQ2 Under what conditions (learning rate, local update count, sampling) can local GD/SGD achieve convergence in non-convex FL settings?
• RQ3 What are the convergence rates for non-convex and PL-conditioned objectives under bounded gradient diversity?
• RQ4 Can the results be extended to networked (neighbor-based) distributed optimization and sampled-device settings?

Key findings

• Local descent with periodic averaging achieves convergence under bounded gradient diversity, with rates matching or improving prior work in various regimes.
• For non-convex objectives under PL, the paper shows improved rates such as O(1/(KT)) dependence compared to some prior bounds.
• Convergence rates hold for both centralized (parameter server) and decentralized networked FL, and for full and stochastic gradient settings.
• Learning-rate and local-update choices depend on gradient diversity, enabling linear speedups when diversity is controlled.
• The analysis reconciles empirical findings by showing variance-reduction-like behavior without explicit variance reduction techniques under proper hyperparameter tuning.