[Paper Review] Federated Learning with Buffered Asynchronous Aggregation

John Nguyen, Kshitiz Malik+5 more|arXiv (Cornell University)|Jun 11, 2021

Privacy-Preserving Technologies in Data61 references37 citations

TL;DR

FedBuff introduces buffered asynchronous aggregation for federated learning, enabling high concurrency while remaining compatible with Secure Aggregation and differential privacy, with proven convergence in smooth non-convex settings and empirical efficiency gains over synchronous and asynchronous baselines.

ABSTRACT

Scalability and privacy are two critical concerns for cross-device federated learning (FL) systems. In this work, we identify that synchronous FL - synchronized aggregation of client updates in FL - cannot scale efficiently beyond a few hundred clients training in parallel. It leads to diminishing returns in model performance and training speed, analogous to large-batch training. On the other hand, asynchronous aggregation of client updates in FL (i.e., asynchronous FL) alleviates the scalability issue. However, aggregating individual client updates is incompatible with Secure Aggregation, which could result in an undesirable level of privacy for the system. To address these concerns, we propose a novel buffered asynchronous aggregation method, FedBuff, that is agnostic to the choice of optimizer, and combines the best properties of synchronous and asynchronous FL. We empirically demonstrate that FedBuff is 3.3x more efficient than synchronous FL and up to 2.5x more efficient than asynchronous FL, while being compatible with privacy-preserving technologies such as Secure Aggregation and differential privacy. We provide theoretical convergence guarantees in a smooth non-convex setting. Finally, we show that under differentially private training, FedBuff can outperform FedAvgM at low privacy settings and achieve the same utility for higher privacy settings.

Motivation & Objective

• Address scalability in cross-device FL by enabling high concurrency without sacrificing privacy.
• Develop a server update mechanism that aggregates updates in a secure buffer before applying them.
• Ensure compatibility with privacy technologies such as Secure Aggregation and differential privacy.
• Provide theoretical convergence guarantees under smooth non-convex objectives.
• Demonstrate empirical efficiency gains over existing synchronous and asynchronous FL methods.

Proposed method

• Propose FedBuff, where client updates are collected asynchronously but server updates occur only after K updates are accumulated in a secure buffer.
• Use a buffer size K to decouple concurrency from server update frequency, with K = 10 identified as a good default.
• Aggregate updates in a secure Trusted Execution Environment or cryptographic buffer to preserve privacy.
• Provide a convergence analysis for FedBuff in the smooth non-convex setting, accounting for local steps Q and staleness.
• Extend FedBuff to support differential privacy via SecAgg and DP-FTRL, enabling GDP privacy guarantees.
• Incorporate practical improvements such as staleness scaling and learning-rate normalization to aid training stability.

Experimental results

Research questions

• RQ1 Can buffered asynchronous aggregation improve scalability of cross-device FL without compromising privacy?
• RQ2 How does FedBuff's buffer size K affect convergence and communication-efficiency at high concurrency?
• RQ3 Is FedBuff compatible with secure aggregation and differential privacy, and how does DP interact with its convergence?
• RQ4 What are the theoretical convergence guarantees for FedBuff in smooth non-convex objectives under asynchronous updates?
• RQ5 How do practical techniques like staleness weighting and LR normalization influence performance?

Key findings

• FedBuff achieves up to 3.8x higher efficiency than the best synchronous baseline in experiments.
• FedBuff outperforms the closest asynchronous method by up to 2.5x in the reported benchmarks.
• A buffer size of K=10 consistently provides strong performance across CelebA, Sent140, and CIFAR-10 tasks.
• FedBuff remains compatible with Secure Aggregation and can be extended to differential privacy with GDP guarantees.
• Under DP, FedBuff can outperform synchronous FedAvgM with amplified DP-SGD at low privacy settings and match performance at higher privacy settings.
• Theoretical results show an ergodic convergence rate bounds under smooth non-convexity for FedBuff.