[Paper Review] One-Shot Federated Learning

Neel Guha, Ameet Talwalkar+1 more|arXiv (Cornell University)|Feb 28, 2019

Privacy-Preserving Technologies in DataComputer Science7 references138 citations

TL;DR

This paper introduces one-shot federated learning that builds a global model from a single communication round using ensemble models and distillation, achieving a 51.5% average relative ROC-AUC gain over local baselines and 90.1% of the unattainable global ideal.

ABSTRACT

We present one-shot federated learning, where a central server learns a global model over a network of federated devices in a single round of communication. Our approach - drawing on ensemble learning and knowledge aggregation - achieves an average relative gain of 51.5% in AUC over local baselines and comes within 90.1% of the (unattainable) global ideal. We discuss these methods and identify several promising directions of future work.

Motivation & Objective

• Motivate a communication-efficient alternative to iterative federated learning by learning a global model in a single round.
• Exploit ensemble learning to aggregate heterogeneous local models when data is non-IID across devices.
• Explore semi-supervised distillation to compress the ensemble and enable privacy-preserving learning.
• Evaluate the proposed methods on real federated datasets to assess practicality and performance.
• Outline future directions for personalization, privacy guarantees, and non-convex models.

Proposed method

• Each device trains a local convex model (kernelized SVM) to completion on its data.
• Central server selects a subset of local models to form an ensemble (k models) using strategies like CV-based performance, data size, or random selection.
• Ensemble predictions are combined by averaging the selected local models.
• In semi-supervised settings, the ensemble is distilled into a smaller model using proxy unlabeled data via a proximity-based distillation in the dual space.
• Distillation minimizes the L2 difference between teacher ensemble predictions and student model on proxy data to obtain a compact f'(x).
• Discusses privacy benefits of distillation for models requiring local access to support vectors.

Experimental results

Research questions

• RQ1 Can one-shot federation using ensemble learning achieve competitive global performance without iterative communication rounds?
• RQ2 How effective are ensemble selection strategies (CV-based, data-based, random) at improving global performance over locally trained models?
• RQ3 Does distillation on unlabeled proxy data preserve or improve ensemble performance while reducing communication?
• RQ4 What are the practical gains and limitations on real federated datasets with non-IID device data?

Key findings

• Ensemble methods consistently outperform the local baseline across evaluated datasets.
• Selected ensembles (via CV, data, or random strategies) generally outperform full ensembles, except on Gleam where devices are fewer.
• Across datasets, the approach achieves an average relative ROC-AUC gain of 51.5% over local baselines.
• The ensemble approach comes within 90.1% of the unattainable global ideal classifier.
• In semi-supervised settings with proxy data, a distilled model can approximate the ensemble performance with relatively few proxy samples.
• Distillation also offers privacy-friendly advantages by reducing the need to share raw local information.