[Paper Review] Private Federated Learning with Domain Adaptation

Daniel Peterson, Pallika Kanani+1 more|arXiv (Cornell University)|Dec 13, 2019

Privacy-Preserving Technologies in DataComputer Science23 references57 citations

TL;DR

The paper proposes FL with per-user domain adaptation (FL+DE) using a mixture of general and private domain experts, improving accuracy under differential privacy relative to standard DP-FL.

ABSTRACT

Federated Learning (FL) is a distributed machine learning (ML) paradigm that enables multiple parties to jointly re-train a shared model without sharing their data with any other parties, offering advantages in both scale and privacy. We propose a framework to augment this collaborative model-building with per-user domain adaptation. We show that this technique improves model accuracy for all users, using both real and synthetic data, and that this improvement is much more pronounced when differential privacy bounds are imposed on the FL model.

Motivation & Objective

• Motivate privacy-preserving collaborative model training across users with diverse domains.
• Introduce a domain adaptation mechanism in federated learning to personalize predictions per user.
• Demonstrate that combining a shared general model with per-user private models via a mixture of experts improves accuracy under differential privacy.
• Show that private domain experts enable noise-free updates for their domain while preserving global learning benefits.

Proposed method

• Formulate a general model M_G trained via federated learning with differential privacy.
• Initialize each private domain model M_Pi as a copy of M_G and train it on private data without DP noise.
• Use a mixture of experts with a gating function alpha_i(x) to combine M_G and M_Pi outputs for each user.
• Train all components via gradient descent, including the gating weights w_i and biases b_i.
• Evaluate both synthetic regression data and a real spam-detection domain adaptation dataset to compare baseline, FL, and FL+DE under different privacy/noise settings.

Experimental results

Research questions

• RQ1 Does per-user domain adaptation in federated learning improve accuracy for diverse user domains?
• RQ2 Can a mixture-of-experts approach mitigate the accuracy loss of differentially private FL?
• RQ3 How do private domain models interact with the general FL model under varying DP noise levels?
• RQ4 What is the impact of domain adaptation on general model generalization when user domains are underrepresented in the global data?

Key findings

• FL+DE achieves the best accuracy across tested scenarios compared to baseline and standard DP-FL.
• Domain adaptation alone provides ~1.3% absolute accuracy gain on a real dataset.
• Under low noise, DP-FL loses more accuracy than FL+DE; with high noise, DP-FL degrades by 13.9% while FL+DE degrades by only 0.8%.
• In synthetic data, FL+DE remains closer to baseline even as DP noise increases, whereas DP-FL can fall below non-collaborative baselines for some users.
• The gating function learns to trust either the general or private model per region of the input space, enabling targeted domain adaptation.