[Paper Review] A Field Guide to Federated Optimization

Jianyu Wang, Zachary Charles+51 more|arXiv (Cornell University)|Jul 14, 2021

Privacy-Preserving Technologies in Data276 references167 citations

TL;DR

This paper provides practical guidelines for formulating, designing, evaluating, and analyzing federated optimization algorithms, with an emphasis on simulation-driven evaluation and real-world considerations. It centers on FedAvg and related techniques as practical anchors while highlighting issues like communication, heterogeneity, and privacy.

ABSTRACT

Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy protection. The distributed learning process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with privacy and system requirements, and other constraints that are not primary considerations in other problem settings. This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance. The goal of this work is not to survey the current literature, but to inspire researchers and practitioners to design federated learning algorithms that can be used in various practical applications.

Motivation & Objective

• Define and motivate federated optimization and its real-world constraints (communication, heterogeneity, privacy, systems).
• Provide practical guidelines for algorithm design, evaluation, and simulation to bridge theory and practice.
• Illustrate representative techniques and how they interact with system constraints.
• Discuss evaluation strategies, datasets, and realistic tuning to infer real-world performance.
• Outline deployment considerations and theoretical tools relevant to federated optimization.

Proposed method

• Present the canonical federated learning problem and the generalized Federated Averaging (FedAvg) algorithm as a practical framework (Algorithm 1).
• Discuss how partial participation, stochastic approximation, and local updates combine to form FedAvg and its variants (FedOpt).
• Review representative techniques to improve performance (momentum/adaptive methods, bias reduction, regularization, alternative aggregation).
• Describe a framework for simulations to evaluate federated optimization algorithms and guide parameter tuning.
• Outline system-level costs and practical deployment considerations, including privacy-compatible protocols.

Experimental results

Research questions

• RQ1 How should federated optimization problems be formulated to reflect data heterogeneity and privacy constraints?
• RQ2 How can we design and evaluate practical FL algorithms that perform well under communication limits and heterogeneous clients?
• RQ3 What are effective techniques beyond vanilla FedAvg to improve convergence and robustness in federated settings?
• RQ4 How should simulations and tuning strategies be conducted to infer real-world performance?
• RQ5 What is the relationship between federation theory and practical deployment in terms of privacy, robustness, and personalization?

Key findings

• FedAvg can be viewed as a generalized FedOpt framework with flexible ClientOpt and ServerOpt components.
• Partial participation, local updates, and stochastic approximation are central to reducing communication while maintaining convergence under heterogeneity.
• Momentum, adaptive methods, bias reduction in local updates, regularization, and alternative aggregation can improve performance over vanilla FedAvg.
• Realistic simulations and careful tuning of client/server learning rates, cohort size, and local steps are essential to infer real-world performance.
• There are diverse problem formulations (composite optimization, probabilistic/posterior views) that motivate different federated optimization algorithms beyond standard ERM formulations.