[Paper Review] PFA: Privacy-preserving Federated Adaptation for Effective Model Personalization

Bingyan Liu, Yao Guo+1 more|arXiv (Cornell University)|Mar 2, 2021

Privacy-Preserving Technologies in Data61 references4 citations

TL;DR

This paper proposes PFA, a privacy-preserving federated adaptation framework that enables collaborative personalization of federated models by grouping clients with similar data distributions using sparsity-based representations. By identifying and adapting the global model across similar clients without sharing raw data, PFA significantly improves personalization performance over local fine-tuning while preserving privacy.

ABSTRACT

Federated learning (FL) has become a prevalent distributed machine learning paradigm with improved privacy. After learning, the resulting federated model should be further personalized to each different client. While several methods have been proposed to achieve personalization, they are typically limited to a single local device, which may incur bias or overfitting since data in a single device is extremely limited. In this paper, we attempt to realize personalization beyond a single client. The motivation is that during FL, there may exist many clients with similar data distribution, and thus the personalization performance could be significantly boosted if these similar clients can cooperate with each other. Inspired by this, this paper introduces a new concept called federated adaptation, targeting at adapting the trained model in a federated manner to achieve better personalization results. However, the key challenge for federated adaptation is that we could not outsource any raw data from the client during adaptation, due to privacy concerns. In this paper, we propose PFA, a framework to accomplish Privacy-preserving Federated Adaptation. PFA leverages the sparsity property of neural networks to generate privacy-preserving representations and uses them to efficiently identify clients with similar data distributions. Based on the grouping results, PFA conducts an FL process in a group-wise way on the federated model to accomplish the adaptation. For evaluation, we manually construct several practical FL datasets based on public datasets in order to simulate both the class-imbalance and background-difference conditions. Extensive experiments on these datasets and popular model architectures demonstrate the effectiveness of PFA, outperforming other state-of-the-art methods by a large margin while ensuring user privacy. We will release our code at: https://github.com/lebyni/PFA.

Motivation & Objective

• To address the limitation of existing personalization methods that fine-tune only on single-client data, which risks overfitting due to small local data volumes.
• To explore federated adaptation as a new paradigm where multiple clients with similar data distributions collaboratively adapt the global model.
• To ensure privacy during adaptation by avoiding raw data exchange, even across cooperating clients.
• To develop a scalable and efficient method for client grouping based on model sparsity, enabling privacy-preserving collaboration.
• To evaluate the framework on realistic, manually constructed FL datasets simulating class imbalance and background differences.

Proposed method

• PFA uses the sparsity pattern of activated neurons in a pre-trained federated model as a privacy-preserving representation for each client.
• These sparsity vectors are computed from the final fully connected layer and used to measure similarity between clients via cosine similarity.
• Clients are grouped based on their sparsity vector similarity, forming clusters of clients with comparable data distributions.
• Within each group, a new federated learning process is conducted to adapt the global model collaboratively, using only model parameters and not raw data.
• The framework ensures differential privacy by allowing noise injection into sparsity vectors, though this is left for future work.
• The approach avoids data leakage by never transmitting raw data or model weights beyond the client and server's final adapted model.

Experimental results

Research questions

• RQ1 Can federated adaptation—collaborative model adaptation across multiple similar clients—significantly improve personalization beyond single-client fine-tuning?
• RQ2 How can client similarity be measured effectively without sharing raw data, especially under strict privacy constraints?
• RQ3 Can sparsity patterns in neural network activations serve as a reliable and privacy-preserving proxy for data distribution similarity?
• RQ4 Does group-wise federated adaptation using sparsity-based clustering lead to better generalization and reduced overfitting compared to local adaptation?
• RQ5 How does PFA perform under realistic non-IID conditions such as class imbalance and background variation?

Key findings

• PFA outperforms state-of-the-art personalization methods, including local fine-tuning and knowledge distillation, by a large margin on both synthetic and real-world simulated FL datasets.
• The use of sparsity-based representations enables accurate client clustering without exposing raw data, achieving high similarity detection accuracy in client grouping.
• Group-wise adaptation significantly reduces overfitting and bias compared to single-client fine-tuning, especially when local data volumes are small.
• The framework maintains strong privacy guarantees by ensuring no raw data or model parameters are shared during the adaptation phase.
• Experiments show that PFA achieves higher test accuracy across diverse non-IID settings, including class-imbalanced and background-varied data distributions.
• The ablation study confirms that sparsity-based client grouping is more effective than random or heuristic-based grouping for personalization.