[論文レビュー] Adaptive Federated Optimization

Sashank J. Reddi, Zachary Charles+6 more|arXiv (Cornell University)|Feb 29, 2020

Privacy-Preserving Technologies in Data参考文献 45被引用数 128

ひとこと要約

この論文は、FedOpt フレームワーク内でフェデレーテッド学習のための適応サーバーサイド最適化手法（FedAdagrad、FedAdam、FedYogi）を提案し、非凸設定の収束解析を提供し、多様なクロスデバイスタスクでの経験的性能とチューニングの容易さを示す。

ABSTRACT

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.

研究の動機と目的

• Address convergence and tuning challenges of FedAvg in heterogeneous federated data.
• Propose a unified FedOpt framework that enables server-side adaptivity.
• Analyze convergence of adaptive server optimization in nonconvex FL settings.
• Empirically validate adaptive federated optimizers across image/text tasks and benchmarks.

提案手法

• General FedOpt framework: server updates apply a gradient-based optimizer to the average client update vector.
• Specialize FedOpt with ServerOpt as adaptive optimizers (Adagrad, Adam, Yogi) and ClientOpt as SGD.
• Provide convergence analyses under nonconvex assumptions with full participation (extendable to partial participation).
• Show that FedAvg is a special case with SGD on client and server and learning rate 1.
• Derive Corollaries illustrating concrete convergence rates and parameter choices (η, η_l, τ).
• Experiment with seven FL tasks across five datasets, comparing FedAdagrad, FedAdam, FedYogi against FedAvg, FedAvgM, and SCAFFOLD.

実験結果

リサーチクエスチョン

• RQ1 Can adaptive server optimization improve convergence in federated learning with heterogeneous data?
• RQ2 How do local (client) updates and server-side adaptivity interact to affect convergence and communication efficiency?
• RQ3 Do adaptive federated optimizers provide easier tuning and better empirical performance in cross-device FL?

主な発見

タスク	FedAdagrad	FedAdam	FedYogi	FedAvgM	FedAvg
CIFAR-10	72.1	77.4	78.0	77.4	72.8
CIFAR-100	47.9	52.5	52.4	52.4	44.7
EMNIST CR	85.1	85.6	85.5	85.2	84.9
Shakespeare	57.5	57.0	57.2	57.3	56.9
SO NWP	23.8	25.2	25.2	23.8	19.5
SO LR	67.1	65.8	65.9	36.9	30.0
EMNIST AE	4.20	1.01	0.98	1.65	6.47

• Adaptive federated optimizers substantially outperform non-adaptive baselines on several tasks, especially in sparse-gradient settings like Stack Overflow NWP and LR.
• FedAdam and FedYogi offer faster initial convergence and easier tuning compared to FedAvgM across most tasks.
• Theoretical results show convergence guarantees for Adagrad, Adam, and Yogi as server optimizers under nonconvex settings, with rates aligning with the best-known nonconvex FL benchmarks.
• Increasing local updates (K) can reduce communication rounds, with trade-offs influenced by client heterogeneity (σ_g).
• Empirical benchmarks and open-source implementation enable reproducible comparison across FL methods.