[Paper Review] CaT: Balanced Continual Graph Learning with Graph Condensation

Yilun Liu, Ruihong Qiu+1 more|arXiv (Cornell University)|Sep 18, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes CaT, a novel framework for balanced continual graph learning that uses graph condensation to create a compact, informative memory bank (Condensed Graph Memory, CGM) and a Training-in-Memory (TiM) strategy to balance model updates. By replacing large incoming graphs with condensed synthetic graphs during training, CaT mitigates data imbalance and catastrophic forgetting, achieving state-of-the-art performance on benchmark datasets with improved efficiency.

ABSTRACT

Continual graph learning (CGL) is purposed to continuously update a graph model with graph data being fed in a streaming manner. Since the model easily forgets previously learned knowledge when training with new-coming data, the catastrophic forgetting problem has been the major focus in CGL. Recent replay-based methods intend to solve this problem by updating the model using both (1) the entire new-coming data and (2) a sampling-based memory bank that stores replayed graphs to approximate the distribution of historical data. After updating the model, a new replayed graph sampled from the incoming graph will be added to the existing memory bank. Despite these methods are intuitive and effective for the CGL, two issues are identified in this paper. Firstly, most sampling-based methods struggle to fully capture the historical distribution when the storage budget is tight. Secondly, a significant data imbalance exists in terms of the scales of the complex new-coming graph data and the lightweight memory bank, resulting in unbalanced training. To solve these issues, a Condense and Train (CaT) framework is proposed in this paper. Prior to each model update, the new-coming graph is condensed to a small yet informative synthesised replayed graph, which is then stored in a Condensed Graph Memory with historical replay graphs. In the continual learning phase, a Training in Memory scheme is used to update the model directly with the Condensed Graph Memory rather than the whole new-coming graph, which alleviates the data imbalance problem. Extensive experiments conducted on four benchmark datasets successfully demonstrate superior performances of the proposed CaT framework in terms of effectiveness and efficiency. The code has been released on https://github.com/superallen13/CaT-CGL.

Motivation & Objective

• To address the data imbalance problem in replay-based continual graph learning (CGL), where large incoming graphs dominate training over small replayed graphs.
• To overcome the limited representational capacity of sampling-based memory banks under tight storage budgets.
• To improve model stability and performance by balancing training between historical and incoming data.
• To propose a plug-and-play solution that enhances existing replay-based CGL methods without architectural changes.

Proposed method

• The Condensed Graph Memory (CGM) is dynamically updated by condensing each new incoming graph into a small, informative synthetic graph using differentiable graph condensation.
• A Training-in-Memory (TiM) scheme replaces full-batch training on incoming graphs with training exclusively on the CGM, ensuring balanced data scale during optimization.
• Graph condensation is applied via differentiable optimization to preserve structural and feature information while minimizing size.
• The framework integrates seamlessly with existing replay-based CGL methods as a plug-and-play enhancement.
• The CGM is maintained under a fixed storage budget, ensuring computational and memory efficiency.
• The method is evaluated across multiple GNN encoders (GCN, SGC) and datasets (CoraFull, Arxiv, Reddit, Products), demonstrating robustness.

Experimental results

Research questions

• RQ1 Can graph condensation effectively replace large incoming graphs in continual graph learning while preserving model performance?
• RQ2 Does training exclusively on a condensed memory bank (TiM) alleviate data imbalance and reduce catastrophic forgetting?
• RQ3 How does the performance of CaT compare to existing replay-based CGL methods under varying storage budgets?
• RQ4 Can the TiM scheme be universally applied to enhance existing replay-based CGL frameworks?
• RQ5 How sensitive is CaT's performance to the choice of graph encoder and memory budget?

Key findings

• CaT achieves state-of-the-art performance on all four benchmark datasets (CoraFull, Arxiv, Reddit, Products) in both task-IL and class-IL settings.
• With a 0.01 budget ratio, CaT achieves 81.7% average performance on Products, significantly outperforming SSM (74.6%) and ER-GNN (42.8%) under the same conditions.
• The TiM scheme improves average performance across all methods: for example, SSM's performance increases from 92.8% to 92.8% on Reddit (with TiM), while CGM with TiM reaches 81.7% on Products.
• Visualizations show that CaT with TiM significantly reduces catastrophic forgetting, as evidenced by more consistent performance matrices across tasks.
• The framework is robust across different graph encoders: GCN and SGC both yield strong results, with GCN slightly outperforming SGC under very low budget conditions.
• Extensive ablation confirms that both CGM and TiM are essential—removing either leads to performance degradation, validating their complementary roles.