[Paper Review] Federated Evaluation of On-device Personalization

Kangkang Wang, Rajiv Mathews+4 more|arXiv (Cornell University)|Oct 22, 2019

Human Mobility and Location-Based AnalysisSocial Sciences117 citations

TL;DR

The paper introduces Federated Personalization Evaluation (FPE) to assess on-device personalization within federated learning, demonstrating privacy-preserving evaluation and notable user-level benefits for a mobile keyboard language model.

ABSTRACT

Federated learning is a distributed, on-device computation framework that enables training global models without exporting sensitive user data to servers. In this work, we describe methods to extend the federation framework to evaluate strategies for personalization of global models. We present tools to analyze the effects of personalization and evaluate conditions under which personalization yields desirable models. We report on our experiments personalizing a language model for a virtual keyboard for smartphones with a population of tens of millions of users. We show that a significant fraction of users benefit from personalization.

Motivation & Objective

• Motivate the need to personalize global models on-device without compromising privacy.
• Extend the federated learning framework to evaluate personalization strategies.
• Identify hyperparameters and gating mechanisms that ensure personalized models improve user experience.
• Demonstrate, via large-scale experiments, that personalization benefits a significant fraction of users.

Proposed method

• Use a CIFG-based LSTM next-word predictor with tied input/output embeddings.
• Train the base global model with Federated Averaging across millions of clients.
• Personalize by fine-tuning from the global model on each device using local data and defined train/test splits.
• Evaluate personalization by comparing metrics on a local test set before and after personalization, and upload delta metrics.
• Aggregate delta metrics across clients to analyze distributions and determine effective hyperparameters.

Experimental results

Research questions

• RQ1 Can on-device personalization improve user experience without harming others, and how can we gate deployments accordingly?
• RQ2 What hyperparameters (batch size, learning rate, stopping criteria) maximize the share of users who benefit from personalization?
• RQ3 How do personalization gains vary with user data characteristics (token counts, baseline accuracy)?
• RQ4 Is it feasible to evaluate personalization within privacy-preserving FL infrastructure at large scale?

Key findings

• Personalization yields mean accuracy improvement from 0.166 to 0.19, a 14.5% relative increase.
• Across 500k+ clients, different hyperparameters yield varying fractions of users with at least 0.02 accuracy improvement (e.g., 47% at B=5, L=0.1; 39% at B=10, L=1.0; 29% at B=10, L=0.1).
• Smaller batch sizes can lead to higher degradation for some users at higher learning rates, visible in the delta histograms.
• Users with worse baseline accuracy tend to benefit more from personalization (e.g., substantial gains for baseline ≤ 0.1).
• The results underscore the importance of data-aware hyperparameter tuning and a gating mechanism to deploy personalized models only when beneficial.