[Paper Review] Lower Bounds and Optimal Algorithms for Personalized Federated Learning

Filip Hanzely, Slavomír Hanzely+2 more|arXiv (Cornell University)|Oct 5, 2020

Privacy-Preserving Technologies in Data55 references64 citations

TL;DR

This paper establishes lower bounds for the personalized federated learning formulation and presents several algorithms that are provably optimal in communication and local computation, including accelerated variants of FedProx and FedAvg/Local SGD.

ABSTRACT

In this work, we consider the optimization formulation of personalized federated learning recently introduced by Hanzely and Richtárik (2020) which was shown to give an alternative explanation to the workings of local { t SGD} methods. Our first contribution is establishing the first lower bounds for this formulation, for both the communication complexity and the local oracle complexity. Our second contribution is the design of several optimal methods matching these lower bounds in almost all regimes. These are the first provably optimal methods for personalized federated learning. Our optimal methods include an accelerated variant of { t FedProx}, and an accelerated variance-reduced version of { t FedAvg}/Local { t SGD}. We demonstrate the practical superiority of our methods through extensive numerical experiments.

Motivation & Objective

• Motivate personalization in federated learning by using a mixing objective that allows local models to differ while penalizing dissimilarity.
• Establish lower bounds on communication and local computation complexity for the personalized FL formulation.
• Develop and analyze optimal algorithms that match these lower bounds in various regimes.

Proposed method

• Study the mixing FL objective F(x) = f(x) + lambda psi(x) and characterize its properties.
• Derive lower bounds on communication rounds and local proximal/gradient/proximal-sum oracle calls.
• Propose and analyze optimal algorithms including Accelerated Proximal Gradient Descent (APGD) variants, Inexact APGD (IAPGD) with AGD or Katyusha as local solvers, and an accelerated L2SGD+ (AL2SGD+) method.
• Show that local solvers can be inexact while preserving optimal rates.
• Compare local and communication complexities across regimes lambda vs L, providing conditions for optimality.
• Provide experimental evidence comparing IAPGD+Katyusha, AL2SGD+, and L2SGD+ across datasets and data distributions.

Experimental results

Research questions

• RQ1 What are the fundamental lower bounds for communication and local computation in personalized federated learning with the mixing objective?
• RQ2 Can we design algorithms that achieve these lower bounds under various oracle models (proximal, gradient, summand gradient)?
• RQ3 How do the optimal rates depend on L (smoothness), mu (strong convexity), and lambda (personalization strength)?
• RQ4 What are the practical performance implications of using inexact local solvers in federated personalization?

Key findings

• The paper proves that to reach an epsilon-neighborhood of the optimum, a minimum of O(sqrt(min{L, lambda}/mu) log(1/epsilon)) communication rounds is required.
• The local proximal/gradient/summand gradient oracle complexities are respectively at least O(sqrt(min{L, lambda}/mu) log(1/epsilon)) or O(sqrt(L/mu) log(1/epsilon)) in corresponding settings.
• Several algorithms achieve these optimal rates in different regimes, including APGD variants and IAPGD with AGD or Katyusha as local solvers, and AL2SGD+ which attains optimal communication and local gradient complexities under broad conditions.
• IAPGD + Katyusha provides optimal communication with near-optimal local gradients under summand-gradient settings, though may incur extra log factors in practice; AL2SGD+ provides universally optimal communication and favorable local gradient complexity.
• The results establish the first provably optimal methods for personalized federated learning and justify the use of local methods in heterogeneous data scenarios.