[Paper Review] FedL2P: Federated Learning to Personalize

Royson Lee, Minyoung Kim+5 more|arXiv (Cornell University)|Oct 3, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

FedL2P proposes a federated meta-learning framework that automatically learns client-specific personalization strategies for federated learning by training meta-networks to adapt batch normalization statistics and layer-wise learning rates based on local data statistics. It improves performance over hand-crafted baselines under both label and feature shift by enabling per-client, scenario-aware personalization through collaborative FL-based meta-learning.

ABSTRACT

Federated learning (FL) research has made progress in developing algorithms for distributed learning of global models, as well as algorithms for local personalization of those common models to the specifics of each client's local data distribution. However, different FL problems may require different personalization strategies, and it may not even be possible to define an effective one-size-fits-all personalization strategy for all clients: depending on how similar each client's optimal predictor is to that of the global model, different personalization strategies may be preferred. In this paper, we consider the federated meta-learning problem of learning personalization strategies. Specifically, we consider meta-nets that induce the batch-norm and learning rate parameters for each client given local data statistics. By learning these meta-nets through FL, we allow the whole FL network to collaborate in learning a customized personalization strategy for each client. Empirical results show that this framework improves on a range of standard hand-crafted personalization baselines in both label and feature shift situations.

Motivation & Objective

• Address the limitation of one-size-fits-all personalization strategies in federated learning under diverse data heterogeneity.
• Overcome the scalability and client-specificity issues of existing personalized FL methods that rely on fixed or manually tuned hyperparameters.
• Enable collaborative learning of personalized hyperparameters across clients without requiring prior knowledge of client data distributions.
• Generalize existing personalization techniques—such as FedBN and selective fine-tuning—by learning their optimal application conditions via meta-learning.
• Improve performance in both label shift and feature shift scenarios through dynamic, data-adaptive personalization strategies

Proposed method

• Train meta-networks (BNNet and LRNet) that map client-specific local data statistics to personalized hyperparameters for batch normalization and learning rates.
• Use federated meta-learning to jointly optimize these meta-networks across clients, enabling collaborative discovery of effective personalization strategies.
• Employ hyper-gradient meta-learning to estimate client-specific hyperparameters without requiring full retraining or manual tuning.
• Integrate BNNet to dynamically balance between client-side and global batch norm statistics, reducing distribution shift impact.
• Use LRNet to infer layer-wise learning rates based on client data characteristics, enabling adaptive fine-tuning depth per layer.
• Apply spectral clustering to evaluate alignment between learned hyperparameter clusters and true data domains, validating the method's ability to capture client-specific patterns

Experimental results

Research questions

• RQ1 Can a unified federated meta-learning framework learn personalized hyperparameters that adapt to both client-specific data distributions and global FL dynamics?
• RQ2 How does FedL2P’s client-specific personalization strategy compare to hand-crafted baselines under label shift and feature shift?
• RQ3 To what extent can the meta-networks capture client-specific heterogeneity in data distributions, such as marginal and conditional shifts?
• RQ4 Does learning personalized hyperparameters through collaborative FL improve generalization and performance across diverse datasets and client types?
• RQ5 Can FedL2P discover and generalize known personalization heuristics (e.g., FedBN, selective fine-tuning) as special cases through data-driven adaptation?

Key findings

• FedL2P outperforms standard hand-crafted personalization baselines in both label shift and feature shift scenarios, demonstrating consistent gains across diverse datasets.
• The framework achieves perfect clustering (ARI = 1) of client hyperparameters on the Office-Caltech-10 dataset, indicating strong alignment with true data domains.
• On DomainNet, both BNNet and LRNet preserve meaningful cluster structures in hyperparameters, with off-diagonal blocks in distance maps showing alignment with true domains despite some domain overlap.
• Abllation studies show that LRNet alone can surpass the performance of both BNNet and LRNet combined on some benchmarks, highlighting the importance of adaptive learning rate scheduling.
• FedL2P generalizes existing methods like FedBN and selective fine-tuning as special cases, learning when and where to apply them based on client-specific data statistics.