[Paper Review] Federated Learning of a Mixture of Global and Local Models

Filip Hanzely, Peter Richtárik|arXiv (Cornell University)|Feb 10, 2020

Privacy-Preserving Technologies in DataComputer Science47 references242 citations

TL;DR

The paper introduces a new federated learning formulation that blends a global model with fully local models through a penalty, and develops Loopless Local Gradient Descent variants with convergence guarantees, showing local steps can reduce communication in heterogeneous data settings and enabling personalization.

ABSTRACT

We propose a new optimization formulation for training federated learning models. The standard formulation has the form of an empirical risk minimization problem constructed to find a single global model trained from the private data stored across all participating devices. In contrast, our formulation seeks an explicit trade-off between this traditional global model and the local models, which can be learned by each device from its own private data without any communication. Further, we develop several efficient variants of SGD (with and without partial participation and with and without variance reduction) for solving the new formulation and prove communication complexity guarantees. Notably, our methods are similar but not identical to federated averaging / local SGD, thus shedding some light on the role of local steps in federated learning. In particular, we are the first to i) show that local steps can improve communication for problems with heterogeneous data, and ii) point out that personalization yields reduced communication complexity.

Motivation & Objective

• Motivate privacy-preserving learning from heterogeneous edge data without aggregating raw data.
• Propose a mixture FL objective that learns personalized local models while staying close to a global consensus.
• Analyze theoretical properties of the optimal solutions and how personalization affects communication.
• Develop efficient SGD-based algorithms (L2GD, L2SGD+) with convergence guarantees and communication complexity bounds.

Proposed method

• Introduce a new FL objective that optimizes over local models x1,...,xn with a penalty term psi(x) that measures deviation from the mean, forming F(x)=f(x)+λψ(x).
• Show that f(x) is the average of local losses fi(xi) and ψ(x) encodes the dispersion of local models; derive gradients and smoothness/strong convexity properties.
• Derive theoretical results linking λ to the closeness of local and global solutions, including that ψ(x(λ)) is non-increasing in λ and f(x(λ)) is non-decreasing.
• Propose Loopless Local Gradient Descent (L2GD), a non-uniform SGD that alternates local GD steps and averaging steps with a probabilistic mechanism to control communication rounds.
• Extend to variance-reduced local SGD (L2SGD+) to achieve linear convergence and better communication performance; discuss generalizations with partial participation and local subsampling.

Experimental results

Research questions

• RQ1 Can a mixed global/local model objective improve personalization while controlling communication in FL with heterogeneous data?
• RQ2 How does the penalty parameter λ affect the convergence to global versus local solutions and the associated communication complexity?
• RQ3 What are the convergence guarantees and optimal communication rounds for Loopless Local GD (L2GD) and its variance-reduced variant?
• RQ4 Do partial participation, local SGD, and variance reduction improve FL performance under heterogeneity?
• RQ5 How do local steps relate to the MAML-like structure of personalized models in FL?

Key findings

• A new mixed FL formulation yields a unique solution x(λ) that transitions from pure local models (λ→0) to the global model (λ→∞).
• The optimal local models satisfy x_i(λ)=x̄(λ)−(1/λ)∇f_i(x_i(λ)) and ∑i ∇f_i(x_i(λ))=0, linking to a MAML-like update.
• L2GD achieves a bound on the expected distance to x(λ) with a tuned sampling probability p*, yielding a finite expected number of communication rounds that interpolates between SGD and GD.
• Corollary: with p*=λ/(L+λ), the expected communications scale as (2λ/(λ+L))*(L/μ) log(1/ε), approaching zero as λ→0 and matching GD's rate as λ→∞.
• L2SGD+ (variance reduction) attains linear convergence and improved communication complexity, with explicit iteration and communication bounds dependent on problem parameters.
• Empirical results on logistic regression with heterogeneous/local data splits corroborate theoretical claims and demonstrate the benefit of variance reduction and personalization in FL.