[Paper Review] pFedSim: Similarity-Aware Model Aggregation Towards Personalized Federated Learning

Jiahao Tan, Yipeng Zhou+3 more|arXiv (Cornell University)|May 25, 2023

Privacy-Preserving Technologies in Data4 citations

TL;DR

pFedSim proposes a personalized federated learning framework that combines model decoupling and similarity-aware aggregation by using classifier distances to identify similar clients, enabling personalized feature extractors without extra communication or privacy risks. It achieves 2.96% higher average accuracy than state-of-the-art baselines on non-IID data.

ABSTRACT

The federated learning (FL) paradigm emerges to preserve data privacy during model training by only exposing clients' model parameters rather than original data. One of the biggest challenges in FL lies in the non-IID (not identical and independently distributed) data (a.k.a., data heterogeneity) distributed on clients. To address this challenge, various personalized FL (pFL) methods are proposed such as similarity-based aggregation and model decoupling. The former one aggregates models from clients of a similar data distribution. The later one decouples a neural network (NN) model into a feature extractor and a classifier. Personalization is captured by classifiers which are obtained by local training. To advance pFL, we propose a novel pFedSim (pFL based on model similarity) algorithm in this work by combining these two kinds of methods. More specifically, we decouple a NN model into a personalized feature extractor, obtained by aggregating models from similar clients, and a classifier, which is obtained by local training and used to estimate client similarity. Compared with the state-of-the-art baselines, the advantages of pFedSim include: 1) significantly improved model accuracy; 2) low communication and computation overhead; 3) a low risk of privacy leakage; 4) no requirement for any external public information. To demonstrate the superiority of pFedSim, extensive experiments are conducted on real datasets. The results validate the superb performance of our algorithm which can significantly outperform baselines under various heterogeneous data settings.

Motivation & Objective

• To address data heterogeneity in federated learning, where non-IID data distributions degrade model performance.
• To improve personalized model accuracy without exposing additional data statistics or incurring high communication costs.
• To develop a method that accurately identifies similar clients using only model parameters, avoiding privacy leaks from external information.
• To combine model decoupling and similarity-based aggregation in a way that enhances both personalization and generalization.
• To achieve high performance with low sensitivity to hyperparameter tuning and minimal computational overhead.

Proposed method

• pFedSim decouples a neural network into a personalized feature extractor and a classifier, with the classifier trained locally on each client.
• Client similarity is measured by the L2 distance between local classifiers, enabling similarity-aware aggregation of feature extractors.
• The framework operates in two phases: a generalization phase using FedAvg for initial training, followed by a personalization phase with iterative refinement.
• In the personalization phase, feature extractors are updated locally and then aggregated only among clients with similar classifiers to improve generalization.
• The method avoids exposing raw data or statistical summaries, relying solely on model parameters to estimate similarity.
• Hyperparameter ρ controls the balance between local and aggregated feature extractor updates, with experiments showing low sensitivity to its value.

Experimental results

Research questions

• RQ1 Can classifier-based similarity estimation improve personalized federated learning accuracy without exposing additional data information?
• RQ2 How does combining model decoupling with similarity-aware aggregation affect model performance under non-IID data settings?
• RQ3 What is the impact of using only model parameters (not data statistics) for client similarity estimation on privacy and communication efficiency?
• RQ4 How sensitive is the proposed method to hyperparameter tuning, particularly ρ?
• RQ5 Can the method maintain high performance even when clients have very few samples, such as in Tiny-ImageNet with 550 samples per client?

Key findings

• pFedSim achieves an average accuracy improvement of 2.96% over state-of-the-art baselines across multiple datasets under non-IID settings.
• On the challenging Tiny-ImageNet dataset with only 550 samples per client, pFedSim improves accuracy by nearly 10% compared to baselines.
• pFedSim significantly outperforms f-FedAP, which uses batch normalization statistics for similarity estimation, due to more accurate classifier-based distance measurement.
• The model accuracy of pFedSim is robust to hyperparameter ρ, showing low sensitivity to its value across experiments.
• pFedSim maintains low communication and computation overhead, as it only exchanges model parameters and does not require external data or statistical summaries.
• In all evaluated settings, pFedSim achieves the highest accuracy, with the best results on CIFAR-10, CINIC-10, Tiny-ImageNet, and EMNIST under both Dirichlet (α=0.1) and less extreme (α=0.5) non-IID distributions.