[Paper Review] Personalised Federated Learning: A Combinational Approach

Sone Kyaw Pye, Han Yu|arXiv (Cornell University)|Aug 22, 2021

Privacy-Preserving Technologies in Data13 references4 citations

TL;DR

This paper proposes a combinational personalization framework for federated learning (FL) that integrates multiple techniques—fine-tuning, mixture-of-experts, multi-task learning, and knowledge distillation—to improve model performance under non-i.i.d. data, differential privacy (DP), and robust aggregation (RA). It demonstrates that combined approaches consistently outperform individual methods and effectively mitigate performance degradation caused by data heterogeneity and privacy-preserving mechanisms.

ABSTRACT

Federated learning (FL) is a distributed machine learning approach involving multiple clients collaboratively training a shared model. Such a system has the advantage of more training data from multiple clients, but data can be non-identically and independently distributed (non-i.i.d.). Privacy and integrity preserving features such as differential privacy (DP) and robust aggregation (RA) are commonly used in FL. In this work, we show that on common deep learning tasks, the performance of FL models differs amongst clients and situations, and FL models can sometimes perform worse than local models due to non-i.i.d. data. Secondly, we show that incorporating DP and RA degrades performance further. Then, we conduct an ablation study on the performance impact of different combinations of common personalization approaches for FL, such as finetuning, mixture-of-experts ensemble, multi-task learning, and knowledge distillation. It is observed that certain combinations of personalization approaches are more impactful in certain scenarios while others always improve performance, and combination approaches are better than individual ones. Most clients obtained better performance with combined personalized FL and recover from performance degradation caused by non-i.i.d. data, DP, and RA.

Motivation & Objective

• To address performance degradation in federated learning due to non-i.i.d. data distributions across clients.
• To investigate the impact of differential privacy (DP) and robust aggregation (RA) on FL model performance.
• To evaluate and compare various personalization techniques—fine-tuning, mixture-of-experts, multi-task learning, and knowledge distillation—individually and in combination.
• To identify optimal combinations of personalization strategies that consistently improve client-specific performance across diverse FL scenarios.
• To demonstrate that combined personalization approaches can recover from performance loss due to data heterogeneity and privacy constraints.

Proposed method

• The authors implement a hybrid personalization framework combining fine-tuning, mixture-of-experts (MoE), multi-task learning (MTL), and knowledge distillation (KD).
• They apply these techniques in a sequential and complementary manner: first training a global FL model, then adapting it per client via fine-tuning and MoE, while leveraging MTL and KD for knowledge transfer.
• The framework integrates differential privacy (DP) and robust aggregation (RA) to preserve privacy and model integrity during training.
• Performance is evaluated across multiple deep learning tasks under varying data non-i.i.d. levels and privacy constraints.
• An ablation study systematically isolates and compares the contribution of each personalization component and their combinations.
• Client-specific performance is measured using accuracy and robustness metrics across diverse data distributions.

Experimental results

Research questions

• RQ1 How does data non-i.i.d. distribution affect the performance of standard federated learning models across clients?
• RQ2 To what extent do differential privacy and robust aggregation degrade FL model performance?
• RQ3 Which individual personalization techniques—fine-tuning, MoE, MTL, or KD—yield the most consistent performance gains?
• RQ4 Do combined personalization strategies outperform individual approaches in heterogeneous and privacy-constrained FL settings?
• RQ5 Can a combinational personalization framework recover performance losses caused by non-i.i.d. data, DP, and RA?

Key findings

• Standard federated learning models often underperform compared to local models due to data non-i.i.d. distribution.
• The application of differential privacy and robust aggregation further degrades FL model performance across clients.
• Combination of personalization techniques consistently improves client-specific performance, with certain combinations showing superior gains in specific scenarios.
• The proposed combinational approach effectively recovers performance losses from non-i.i.d. data, DP, and RA, outperforming both global and individual personalized models.
• Fine-tuning combined with knowledge distillation and mixture-of-experts yields the most robust performance improvements across diverse client data distributions.
• Certain combinations of personalization methods are universally beneficial, while others are context-dependent, highlighting the need for adaptive strategy selection.