[Paper Review] Papaya: Practical, Private, and Scalable Federated Learning

Dzmitry Huba, John Nguyen+12 more|arXiv (Cornell University)|Nov 8, 2021

Privacy-Preserving Technologies in DataComputer Science30 references29 citations

TL;DR

Papaya presents AsyncFL, a production-grade federated learning system that enables asynchronous secure aggregation and outperforms synchronous FL in speed and communication efficiency at scale. It demonstrates near-linear server update frequency with concurrency and reduced bias compared to over-selection in SyncFL.

ABSTRACT

Cross-device Federated Learning (FL) is a distributed learning paradigm with several challenges that differentiate it from traditional distributed learning, variability in the system characteristics on each device, and millions of clients coordinating with a central server being primary ones. Most FL systems described in the literature are synchronous - they perform a synchronized aggregation of model updates from individual clients. Scaling synchronous FL is challenging since increasing the number of clients training in parallel leads to diminishing returns in training speed, analogous to large-batch training. Moreover, stragglers hinder synchronous FL training. In this work, we outline a production asynchronous FL system design. Our work tackles the aforementioned issues, sketches of some of the system design challenges and their solutions, and touches upon principles that emerged from building a production FL system for millions of clients. Empirically, we demonstrate that asynchronous FL converges faster than synchronous FL when training across nearly one hundred million devices. In particular, in high concurrency settings, asynchronous FL is 5x faster and has nearly 8x less communication overhead than synchronous FL.

Motivation & Objective

• Motivate the need for scalable cross-device FL that handles heterogeneity and stragglers.
• Propose an asynchronous FL design (AsyncFL) with buffered secure aggregation for zero-wait client updates.
• Demonstrate production-scale evaluation on millions of devices to measure convergence, throughput, and fairness.
• Address design challenges like client independence, high utilization, and fast model aggregation.

Proposed method

• Describe AsyncFL algorithm (FedBuff) with no rounds and aggregation goal-driven updates.
• Introduce asynchronous secure aggregation using a Trusted Execution Environment to mask and unmask updates.
• Present a two-tier system design (Coordinator, Selector, Aggregator) enabling independent client participation.
• Explain a fast, parallelized aggregation pipeline with persistent aggregators and in-memory queues.
• Detail a client selection and replacement mechanism to sustain near-100% utilization.
• Provide a traffic- and scale-aware benchmarking methodology in a production-like setting.

Experimental results

Research questions

• RQ1 How does asynchronous federated learning (AsyncFL) compare to synchronous FL (SyncFL) in convergence speed at large scale?
• RQ2 Can asynchronous secure aggregation maintain privacy while enabling high utilization and low bias under heterogeneity?
• RQ3 What system design choices enable production-scale AsyncFL on millions of devices?

Key findings

• AsyncFL converges faster than SyncFL, achieving up to 5x wall-clock speedup in high concurrency settings.
• AsyncFL achieves up to 8x reduction in communication overhead compared to SyncFL.
• AsyncFL generates up to 30x more server model updates per unit time than SyncFL.
• Over-selection in SyncFL introduces sampling bias against slow devices and high-data clients, reducing model fairness.
• AsyncFL maintains low bias similar to unskewed SyncFL while delivering faster training and higher throughput.
• AsyncFL demonstrates improved fairness, with slow devices not disproportionately dropped.