[Paper Review] Towards Federated Learning at Scale: System Design

Keith Bonawitz, Hubert Eichner+12 more|arXiv (Cornell University)|Feb 4, 2019

Privacy-Preserving Technologies in Data21 references954 citations

TL;DR

The paper presents a production-scale system for federated learning on mobile devices using TensorFlow, detailing the high-level architecture, protocol, device and server design, secure aggregation, analytics, tooling, and production experience. It discusses challenges, solutions, and open issues for scaling Federated Averaging in real-world deployments.

ABSTRACT

Federated Learning is a distributed machine learning approach which enables model training on a large corpus of decentralized data. We have built a scalable production system for Federated Learning in the domain of mobile devices, based on TensorFlow. In this paper, we describe the resulting high-level design, sketch some of the challenges and their solutions, and touch upon the open problems and future directions.

Motivation & Objective

• Motivate and define the challenges of deploying Federated Learning at scale on mobile devices.
• Describe the end-to-end system architecture, including device, server, and protocol components.
• Explain how synchronous rounds and Federated Averaging are implemented with practical scalability and reliability.
• Discuss privacy enhancements (e.g., Secure Aggregation) and data-security considerations.
• Outline tooling, analytics, and deployment experiences to guide future systems research.

Proposed method

• Define the federated learning protocol with participants (devices) and a cloud server.
• Describe a three-phase round (Selection, Configuration, Reporting) and pace steering for scalable participation.
• Explain device architecture, example stores, and on-device FL runtime integration via Android’s IPC (AIDL).
• Detail server architecture using the Actor Model with Coordinators, Selectors, Master Aggregators, and Aggregators for scalable in-memory, ephemeral state and parallel aggregation.
• Discuss Secure Aggregation as a privacy-enhancing addition and its four-round protocol.
• Present analytics and operational tooling to monitor device health and system performance.
• Outline model engineer workflow, including modeling/simulation, plan generation, versioning, testing, and deployment.
• Highlight applications (e.g., on-device next-word prediction, on-device item ranking) and empirical production metrics.

Experimental results

Research questions

• RQ1 How can federated learning be scaled to tens of millions of devices with reliable synchronous rounds?
• RQ2 What architectural choices enable efficient server-side aggregation and device coordination at large scale?
• RQ3 How can privacy be enhanced (e.g., Secure Aggregation) without compromising scalability and performance?
• RQ4 What tooling, workflows, and analytics are needed to develop, test, and deploy FL tasks on devices?
• RQ5 What is the practical performance and deployment experience of FL in production (e.g., 10M+ devices, billions potential)?

Key findings

• The system supports training a deep neural network on-device with updates aggregated in the cloud via Federated Averaging, using Secure Aggregation to protect individual updates.
• The server design using an Actor Model enables scalable, in-memory, ephemeral state and dynamic resource management for rounds across populations from tens to potentially billions of devices.
• Pace steering and carefully designed selection/reporting windows help scale rounds and mitigate thundering herd problems while accommodating diurnal device activity.
• Empirical production experience shows up to 10k devices participate concurrently, with about 6-10% device dropout and a typical initial participation overshoot of 130% to compensate for stragglers.
• Plan versioning and automated testing in simulation address on-device TensorFlow version heterogeneity, enabling safe deployment of FL plans across device fleets.
• On-device next-word prediction and on-device ranking tasks demonstrate competitive performance gains and practical convergence within FL rounds, with production insights on wall-clock efficiency.