[Paper Review] Three Approaches for Personalization with Applications to Federated Learning

Yishay Mansour, Mehryar Mohri+2 more|arXiv (Cornell University)|Feb 25, 2020

Privacy-Preserving Technologies in Data55 references320 citations

TL;DR

The paper develops a learning-theoretic framework for personalization in federated learning and analyzes three practical approaches—user clustering, data interpolation, and model interpolation—with guarantees and efficient algorithms, plus empirical validation.

ABSTRACT

The standard objective in machine learning is to train a single model for all users. However, in many learning scenarios, such as cloud computing and federated learning, it is possible to learn a personalized model per user. In this work, we present a systematic learning-theoretic study of personalization. We propose and analyze three approaches: user clustering, data interpolation, and model interpolation. For all three approaches, we provide learning-theoretic guarantees and efficient algorithms for which we also demonstrate the performance empirically. All of our algorithms are model-agnostic and work for any hypothesis class.

Motivation & Objective

• Motivate personalization as an intermediate alternative to purely local or global models in FL.
• Provide learning-theoretic guarantees (generalization bounds) for each personalization approach.
• Develop efficient, communication- and computation-friendly algorithms for clustering, data interpolation, and model interpolation.
• Demonstrate empirical performance improvements on synthetic data and the EMNIST dataset.

Proposed method

• Propose three orthogonal personalization strategies: user clustering (HypCluster), data interpolation (Dapper), and model interpolation (Mapper).
• For clustering, define hypothesis-based clustering and derive generalization bounds; present HypCluster as an EM-type stochastic algorithm.
• For data interpolation, formulate a lambda-weighted data mixture between local and cluster/global data and derive generalization guarantees; introduce Dapper with provable convergence under strong convexity.
• For model interpolation, learn central and local models and optimize interpolated predictions across clients; provide a theoretical bound on empirical and true losses (Theorem 6.1 and Corollaries).
• Provide efficiency analyses focusing on communication cost, privacy considerations, and scalability to FL settings.

Experimental results

Research questions

• RQ1 How can personalization be achieved with intermediate models between local and global ones while maintaining communication efficiency?
• RQ2 What are the generalization guarantees for clustering-based, data-interpolation, and model-interpolation personalization strategies?
• RQ3 How can we design scalable algorithms that perform well under FL constraints and data heterogeneity?
• RQ4 What empirical gains do these approaches provide on realistic federated datasets?

Key findings

• Three personalization approaches yield theoretical generalization guarantees and practical algorithms for FL.
• HypCluster (user clustering) offers favorable generalization bounds and strong empirical performance, especially with a small number of clusters.
• Data interpolation (Dapper) achieves efficient communication and provable convergence with a tunable lambda between local and global data.
• Model interpolation (Mapper) provides competitive performance with a principled interpolation between central and local models and accompanying bounds.
• In EMNIST experiments, HypCluster with two clusters significantly outperforms FedAvg and Agnostic baselines, with improvements of at least 4.3% in accuracy; further gains of about 1% are achieved by Dapper/Mapper variants.