[Paper Review] Oort: Efficient Federated Learning via Guided Participant Selection

Fan Lai, Xiangfeng Zhu+2 more|arXiv (Cornell University)|Oct 12, 2020

Privacy-Preserving Technologies in DataComputer Science68 references47 citations

TL;DR

Oort introduces guided participant selection for federated learning to improve time-to-accuracy and testing efficiency by prioritizing high-utility, fast participants while respecting developer testing criteria and privacy.

ABSTRACT

Federated Learning (FL) is an emerging direction in distributed machine learning (ML) that enables in-situ model training and testing on edge data. Despite having the same end goals as traditional ML, FL executions differ significantly in scale, spanning thousands to millions of participating devices. As a result, data characteristics and device capabilities vary widely across clients. Yet, existing efforts randomly select FL participants, which leads to poor model and system efficiency. In this paper, we propose Oort to improve the performance of federated training and testing with guided participant selection. With an aim to improve time-to-accuracy performance in model training, Oort prioritizes the use of those clients who have both data that offers the greatest utility in improving model accuracy and the capability to run training quickly. To enable FL developers to interpret their results in model testing, Oort enforces their requirements on the distribution of participant data while improving the duration of federated testing by cherry-picking clients. Our evaluation shows that, compared to existing participant selection mechanisms, Oort improves time-to-accuracy performance by 1.2x-14.1x and final model accuracy by 1.3%-9.8%, while efficiently enforcing developer-specified model testing criteria at the scale of millions of clients.

Motivation & Objective

• Motivate and address the inefficiencies caused by random participant selection in large-scale FL.
• Develop a framework to select participants that optimizes both statistical utility and system speed for training.
• Enable scalable enforcement of developer-specified data distribution criteria during federated testing.
• Respect client privacy in utility estimation and robustness to outliers and staleness in participant utilities.

Proposed method

• Define client statistical utility using a pragmatic approximation based on local training loss and data size.
• Formulate a combined utility to balance statistical and system efficiency, incorporating a pacer to adapt round duration.
• Propose an online exploration-exploitation strategy (bandit-inspired) to select high-utility participants at scale while handling staleness and outliers.
• Provide two selectors: a training selector to improve time-to-accuracy and a testing selector to enforce data distribution criteria during federated testing.
• Integrate Oort with PySyft and evaluate on real-world FL workloads to quantify improvements over state-of-the-art methods.

Experimental results

Research questions

• RQ1 How can FL participants be selected to maximize time-to-accuracy by balancing statistical utility and system speed?
• RQ2 How can developer-specified data distribution criteria be efficiently enforced during federated testing at scale without exposing raw data?
• RQ3 How can privacy be preserved in estimating participant utility for training and testing in FL?
• RQ4 How can the selection mechanism handle utility staleness and outliers in a scalable, robust manner?
• RQ5 Can Oort's approach improve training time-to-accuracy and final model accuracy compared to existing selection methods?

Key findings

• Time-to-accuracy improvements of 1.2x to 14.1x over existing selection methods for federated training.
• Final model accuracy improvements of 1.3% to 9.8% over baselines.
• Federated testing duration reduced by about 4.7x on average when enforcing developer-specified data distributions.
• Oort achieves close to upper-bound statistical performance while scaling to millions of clients.
• The framework supports diverse criteria (e.g., fairness) and maintains privacy through aggregate/noisy utility, with robust performance under noisy signals.