[Paper Review] Federated Continual Learning via Knowledge Fusion: A Survey

Xin Yang, Hao Yu+4 more|arXiv (Cornell University)|Dec 27, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This survey introduces Federated Continual Learning (FCL) via knowledge fusion to address spatial-temporal catastrophic forgetting in decentralized, streaming data environments. It proposes two generic frameworks—synchronous and asynchronous FCL—and categorizes existing methods by knowledge fusion mechanisms, offering a unified perspective on preserving past knowledge while enabling continual learning across heterogeneous clients.

ABSTRACT

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.

Motivation & Objective

• To identify and formalize the fundamental challenge of spatial-temporal catastrophic forgetting in Federated Continual Learning (FCL), combining temporal forgetting on clients and spatial forgetting during global aggregation.
• To integrate existing FCL methods into two generic system frameworks: synchronous and asynchronous FCL, enabling structured analysis and comparison.
• To categorize FCL approaches based on their knowledge fusion mechanisms, such as parameter distillation, replay, and model pruning, to clarify design principles.
• To provide a comprehensive outlook on open challenges and future research directions in FCL, particularly regarding rational collaboration, convergence efficiency, and integration with large models.

Proposed method

• Proposes a dual forgetting framework: temporal catastrophic forgetting (on local clients during sequential task learning) and spatial catastrophic forgetting (during global model aggregation across heterogeneous clients).
• Introduces two generic FCL system architectures: synchronous FCL (centralized coordination) and asynchronous FCL (decentralized, event-driven updates), to model different communication patterns.
• Classifies FCL methods by knowledge fusion mechanisms, including knowledge distillation, experience replay, parameter regularization, and model compression, to highlight design diversity.
• Employs federated averaging (FedAvg) as a baseline to empirically evaluate the impact of spatial-temporal forgetting on model performance.
• Analyzes privacy and efficiency trade-offs in FCL, emphasizing communication efficiency and data quality in client participation.
• Reviews emerging applications of FCL in recommendation systems, large language models (LLMs), and blockchain-based systems to demonstrate real-world viability.

Figure 1: Experiment Results. Solid lines represent knowledge retention of clients in time, and dashed lines represent knowledge retention of clients in space. When the data distribution among clients is identical, and the distribution among local tasks is also the same, spatial-temporal catastrophi

Experimental results

Research questions

• RQ1 How does spatial-temporal catastrophic forgetting emerge in Federated Continual Learning, and what is its impact on model performance?
• RQ2 What are the key architectural and operational differences between synchronous and asynchronous FCL frameworks, and how do they affect knowledge fusion?
• RQ3 How do existing FCL methods mitigate both temporal and spatial forgetting through knowledge fusion mechanisms?
• RQ4 What are the critical challenges in rational collaboration, convergence speed, and data quality in FCL, and how do they affect model efficiency and privacy?
• RQ5 How can FCL be effectively integrated with large language models and other emerging AI systems to support continual, privacy-preserving learning?

Key findings

• Spatial-temporal catastrophic forgetting significantly degrades model performance in FCL, with federated averaging (FedAvg) showing substantial forgetting on previous tasks due to parameter overwriting during aggregation.
• The integration of FCL methods into synchronous and asynchronous frameworks reveals distinct trade-offs in convergence speed, communication overhead, and forgetting mitigation effectiveness.
• Knowledge fusion mechanisms such as knowledge distillation and experience replay are effective in reducing forgetting, but their performance varies under non-IID data and dynamic client participation.
• Data heterogeneity and unreliable client contributions lead to spatial catastrophic forgetting, where global model aggregation fails to preserve task-specific knowledge from individual clients.
• Convergence time in FCL remains a major bottleneck, especially when using rehearsal or generative replay, due to repeated communication and computation across multiple rounds.
• Emerging applications in LLMs, recommendation systems, and blockchain show promise, but FCL integration with large models remains underexplored and limited in current research.

Figure 2: Illustration of synchronous federated continual learning scenario , in which all clients share a common task sequence. Different colors represent different tasks in a task sequence. Apparently, “synchronous” means that clients will process the same task in a shared task order, and the prog