[Paper Review] Federated User Representation Learning

Duc Viet Bui, Kshitiz Malik+5 more|arXiv (Cornell University)|Sep 27, 2019

Privacy-Preserving Technologies in Data27 references37 citations

TL;DR

FURL splits model parameters into federated and private parts, keeping user embeddings on-device to protect privacy, while achieving near-centralized personalization performance in FL.

ABSTRACT

Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propose Federated User Representation Learning (FURL), a simple, scalable, privacy-preserving and resource-efficient way to utilize existing neural personalization techniques in the Federated Learning (FL) setting. FURL divides model parameters into federated and private parameters. Private parameters, such as private user embeddings, are trained locally, but unlike federated parameters, they are not transferred to or averaged on the server. We show theoretically that this parameter split does not affect training for most model personalization approaches. Storing user embeddings locally not only preserves user privacy, but also improves memory locality of personalization compared to on-server training. We evaluate FURL on two datasets, demonstrating a significant improvement in model quality with 8% and 51% performance increases, and approximately the same level of performance as centralized training with only 0% and 4% reductions. Furthermore, we show that user embeddings learned in FL and the centralized setting have a very similar structure, indicating that FURL can learn collaboratively through the shared parameters while preserving user privacy.

Motivation & Objective

• Motivate collaborative personalization and the privacy challenges in FL.
• Introduce a parameter-splitting approach that keeps private user embeddings on-device.
• Provide conditions under which split personalization does not hurt performance.
• Empirically show FL can match centralized personalization with privacy preserved.

Proposed method

• Define federated vs private parameters in a neural personalization model.
• Prove the split-personalization constraints (independent local training and independent aggregation) that guarantee no performance loss.
• Describe the FURL training workflow with Federated Averaging and private parameter updates.
• Show that private parameters can be locally trained and simply retained on-device during aggregation.
• Evaluate using two document classification datasets with LSTM-based models and user embeddings.
• Demonstrate a simple, scalable approach with minimal changes to standard FL.

Experimental results

Research questions

• RQ1 Can personalization techniques that rely on private user embeddings be adapted to FL without sacrificing performance?
• RQ2 Under what conditions does splitting parameters into federated and private parts preserve model quality?
• RQ3 How does FURL compare to centralized training in accuracy and privacy preservation?
• RQ4 Do user embeddings learned in FL resemble those learned centrally in structure?

Key findings

• FURL yields 8% and 51% improvements from personalization on the Sticker and Subreddit datasets, respectively.
• In FL, FURL achieves performance close to centralized training with only 0% and 4% reductions on the same datasets.
• User embeddings learned in FL exhibit similar structures to those learned in centralized training, as shown by embedding visualizations where similar users cluster together.
• Personalization via private embeddings significantly enhances performance in both server-based and FL settings.
• Global aggregation time in FURL scales linearly with the number of users, which is favorable compared to quadratic scaling in some alternative approaches.
• The training process preserves privacy by keeping private embeddings on-device while sharing only federated parameters.