[Paper Review] Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data

Eunjeong Jeong, Seungeun Oh+4 more|arXiv (Cornell University)|Nov 28, 2018

Privacy-Preserving Technologies in DataComputer Science4 references443 citations

TL;DR

The paper introduces Federated Distillation (FD) and Federated Augmentation (FAug) to enable communication-efficient on-device learning under non-IID private data, achieving similar accuracy to federated learning (FL) with far less communication. FD uses per-label mean logit vectors to perform online knowledge distillation, while FAug uses a server-trained GAN to augment data toward IID distributions.

ABSTRACT

On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.

Motivation & Objective

• Motivate and enable on-device ML with privacy-preserving data without heavy communication overhead.
• Address non-IID data across devices that degrades FL performance.
• Propose FD for low-communication knowledge distillation and FAug to IID-ify data distributions.

Proposed method

• FD exchanges only model outputs (logits) rather than full model parameters to enable large local models.
• FD stores per-label mean logit vectors locally and uploads them to compute global-average logits per label, which act as teachers for distillation.
• FAug trains a conditional GAN at a server to augment missing target-label data, enabling devices to locally reproduce IID-like datasets.
• FAug privacy handling includes adding redundant labels to reduce device-server privacy leakage, with inter-device privacy leakage analyzed.
• Evaluation compares FD/FD+FAug with FL/FL+FAug on non-IID MNIST under various device counts and target/redundant label settings.

Experimental results

Research questions

• RQ1 Can FD achieve substantially lower communication than FL while maintaining competitive accuracy on non-IID private data?
• RQ2 Does FAug effectively mitigate non-IID performance gaps with manageable privacy and communication costs?
• RQ3 How do the number of devices, target labels, and redundant labels affect accuracy and privacy leakage in FD/FAug?
• RQ4 What is the trade-off between communication overhead and accuracy when combining FD with FAug versus FL?

Key findings

• FD with FAug achieves around 26x lower total communication than FL while reaching 95-98% test accuracy.
• FD+FAug yields comparable per-label accuracy improvements and overall test accuracy across varying device counts compared to FL.
• FD without FAug under non-IID data shows noticeable accuracy gaps relative to FL, but FAug closes much of this gap (e.g., FD+FAug approaches FL accuracy).
• Inter-device privacy leakage decreases as the number of redundant labels increases and with more devices, reducing the risk of target-label disclosure.
• Device-server privacy leakage decreases with more redundant labels and increases with more target labels.