[Paper Review] Adaptive Federated Optimization

Sashank J. Reddi, Zachary Charles+6 more|arXiv (Cornell University)|Feb 29, 2020

Privacy-Preserving Technologies in Data45 references128 citations

TL;DR

This paper introduces adaptive server-side optimizers for federated learning (FedAdagrad, FedAdam, FedYogi) within a FedOpt framework, provides convergence analysis for nonconvex settings, and demonstrates superior empirical performance and easier tuning across diverse cross-device tasks."

ABSTRACT

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.

Motivation & Objective

• Address convergence and tuning challenges of FedAvg in heterogeneous federated data.
• Propose a unified FedOpt framework that enables server-side adaptivity.
• Analyze convergence of adaptive server optimization in nonconvex FL settings.
• Empirically validate adaptive federated optimizers across image/text tasks and benchmarks.

Proposed method

• General FedOpt framework: server updates apply a gradient-based optimizer to the average client update vector.
• Specialize FedOpt with ServerOpt as adaptive optimizers (Adagrad, Adam, Yogi) and ClientOpt as SGD.
• Provide convergence analyses under nonconvex assumptions with full participation (extendable to partial participation).
• Show that FedAvg is a special case with SGD on client and server and learning rate 1.
• Derive Corollaries illustrating concrete convergence rates and parameter choices (η, η_l, τ).
• Experiment with seven FL tasks across five datasets, comparing FedAdagrad, FedAdam, FedYogi against FedAvg, FedAvgM, and SCAFFOLD.

Experimental results

Research questions

• RQ1 Can adaptive server optimization improve convergence in federated learning with heterogeneous data?
• RQ2 How do local (client) updates and server-side adaptivity interact to affect convergence and communication efficiency?
• RQ3 Do adaptive federated optimizers provide easier tuning and better empirical performance in cross-device FL?

Key findings

• Adaptive federated optimizers substantially outperform non-adaptive baselines on several tasks, especially in sparse-gradient settings like Stack Overflow NWP and LR.
• FedAdam and FedYogi offer faster initial convergence and easier tuning compared to FedAvgM across most tasks.
• Theoretical results show convergence guarantees for Adagrad, Adam, and Yogi as server optimizers under nonconvex settings, with rates aligning with the best-known nonconvex FL benchmarks.
• Increasing local updates (K) can reduce communication rounds, with trade-offs influenced by client heterogeneity (σ_g).
• Empirical benchmarks and open-source implementation enable reproducible comparison across FL methods.