[Paper Review] Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning

Shaoxiong Ji, Yue Tan+7 more|arXiv (Cornell University)|Feb 25, 2021

Privacy-Preserving Technologies in Data229 references19 citations

TL;DR

This paper surveys emerging trends in federated learning, focusing on model fusion techniques—such as adaptive aggregation, regularization, clustering, and Bayesian methods—to address statistical heterogeneity and improve robustness. It further introduces 'Federated X Learning' by integrating FL with paradigms like meta-learning, transfer learning, and unsupervised learning, offering a unified taxonomy and identifying key challenges and future directions in privacy-preserving, scalable, and personalized AI systems.

ABSTRACT

Federated learning is a new learning paradigm that decouples data collection and model training via multi-party computation and model aggregation. As a flexible learning setting, federated learning has the potential to integrate with other learning frameworks. We conduct a focused survey of federated learning in conjunction with other learning algorithms. Specifically, we explore various learning algorithms to improve the vanilla federated averaging algorithm and review model fusion methods such as adaptive aggregation, regularization, clustered methods, and Bayesian methods. Following the emerging trends, we also discuss federated learning in the intersection with other learning paradigms, termed federated X learning, where X includes multitask learning, meta-learning, transfer learning, unsupervised learning, and reinforcement learning. In addition to reviewing state-of-the-art studies, this paper also identifies key challenges and applications in this field, while also highlighting promising future directions.

Motivation & Objective

• Address statistical heterogeneity in federated learning due to non-IID client data distributions.
• Improve model aggregation robustness through advanced fusion techniques like adaptive weighting, regularization, and clustering.
• Explore integration of federated learning with diverse learning paradigms (e.g., meta-learning, transfer learning, unsupervised learning) to enable broader real-world applicability.
• Identify key challenges such as label scarcity, communication efficiency, and client fairness in federated settings.
• Propose a novel taxonomy of federated learning based on model fusion principles and cross-paradigm integration.

Proposed method

• Categorize model fusion methods into four subclasses: adaptive/attentive aggregation, regularization, clustering, and Bayesian methods.
• Introduce a unified taxonomy of federated learning based on model fusion mechanisms and integration with other learning paradigms (e.g., Federated Meta-Learning, Federated Transfer Learning).
• Analyze client contribution weighting using metrics such as data quality, model consistency, and historical performance to improve aggregation robustness.
• Apply unsupervised and semi-supervised techniques to mitigate label scarcity by leveraging unlabeled data and pseudo-labeling.
• Utilize graph-based methods (e.g., FedGP) to correct noisy labels and reduce drift in medical imaging applications.
• Propose on-device personalization via model interpolation and meta-learning to enhance individual client performance without compromising privacy.

Experimental results

Research questions

• RQ1 How can federated model aggregation be made more robust to statistical heterogeneity and non-IID data distributions?
• RQ2 What are the most effective model fusion techniques—such as adaptive weighting, regularization, and clustering—for improving FL convergence and accuracy?
• RQ3 How can federated learning be effectively combined with other learning paradigms like meta-learning, transfer learning, and unsupervised learning?
• RQ4 What are the key challenges in federated learning related to label scarcity, communication cost, and client fairness?
• RQ5 How can on-device personalization be achieved without compromising data privacy or requiring centralized retraining?

Key findings

• Adaptive and attentive aggregation methods such as FedAtt and FedAMP significantly improve model accuracy by dynamically weighting client contributions based on model quality and data characteristics.
• Regularization and clustering-based aggregation reduce model divergence and improve performance on non-IID data, particularly in scenarios with high statistical heterogeneity.
• Federated X learning frameworks—such as Federated Meta-Learning and Federated Transfer Learning—enable faster adaptation and better generalization across clients with limited labeled data.
• Graph-based methods like FedGP enhance robustness in medical imaging by purifying noisy labels and reducing gradient drift through graph-guided loss functions.
• On-device personalization via model interpolation and meta-learning enables clients to achieve higher accuracy on their private data while maintaining global model utility.
• Unsupervised and semi-supervised learning techniques are critical for mitigating label scarcity, enabling effective training in real-world settings where labeling is costly or infeasible.