[Paper Review] Asynchronous Federated Optimization

Cong Xie, Oluwasanmi Koyejo+1 more|arXiv (Cornell University)|Mar 10, 2019

Privacy-Preserving Technologies in Data23 references442 citations

TL;DR

The paper introduces FedAsync, an asynchronous federated optimization algorithm that combines local regularized SGD updates with adaptive mixing to achieve near-linear convergence under certain conditions, tolerating stale updates.

ABSTRACT

Federated learning enables training on a massive number of edge devices. To improve flexibility and scalability, we propose a new asynchronous federated optimization algorithm. We prove that the proposed approach has near-linear convergence to a global optimum, for both strongly convex and a restricted family of non-convex problems. Empirical results show that the proposed algorithm converges quickly and tolerates staleness in various applications.

Motivation & Objective

• Motivate federated learning for edge devices with asynchronous training to improve scalability and responsiveness.
• Propose a new asynchronous federated optimization algorithm that tolerates stale updates.
• Establish convergence guarantees for a restricted class of non-convex problems.
• Develop practical strategies to control error due to asynchrony via adaptive mixing.
• Demonstrate empirical benefits over synchronous FedAvg in realistic settings.

Proposed method

• Propose FedAsync where servers update the global model via weighted averaging with a mixing parameter alpha.
• Each device solves a regularized local problem using SGD on augmented loss f(x;z)+ (rho/2)||x - x_t||^2.
• Allow asynchronous server-worker communication with non-blocking updates and time-stamped stale models.
• Introduce an adaptive mixing function alpha_t = alpha * s(tau difference) to mitigate staleness effects.
• Provide convergence analysis under L-smoothness and mu-weak convexity assumptions with bounded delay and local update counts.
• Derive conditions on rho, gamma, and the staleness to guarantee convergence to a critical point.

Experimental results

Research questions

• RQ1 Can asynchronous federated optimization converge to a global or critical point for nonconvex objectives under bounded delay and heterogenous local updates?
• RQ2 How does staleness affect convergence, and can adaptive mixing mitigate its impact in FedAsync?
• RQ3 Does FedAsync offer practical performance gains over synchronous FedAvg in common datasets under non-IID data?
• RQ4 What mechanisms (regularization, mixing) are sufficient to ensure convergence and stability in asynchronous federated settings?

Key findings

• FedAsync achieves near-linear convergence to a global optimum for strongly convex and a restricted family of non-convex problems under specified assumptions.
• Adaptive mixing strategies improve robustness to staleness and often outperform constant mixing in experiments.
• Empirical results show FedAsync converges quickly and tolerates varying staleness, sometimes beating synchronous FedAvg.
• Larger staleness slows convergence but is not catastrophic when adaptive alpha is used.
• The proposed approach provides a tunable trade-off between convergence rate and variance reduction via the mixing hyperparameter.