[Paper Review] Towards Open Temporal Graph Neural Networks

Kaituo Feng, Changsheng Li+2 more|arXiv (Cornell University)|Mar 27, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes OTGNet, a novel class-incremental learning framework for open temporal graph neural networks that addresses heterophily propagation and catastrophic forgetting. By disentangling node representations into class-agnostic and class-relevant components and using an information bottleneck-based message passing mechanism, OTGNet avoids conflicting feature aggregation across different classes. It further selects diverse and important triad subgraphs for rehearsal, achieving state-of-the-art performance on Reddit, Yelp, and Taobao datasets with significant gains in accuracy and reduced forgetting.

ABSTRACT

Graph neural networks (GNNs) for temporal graphs have recently attracted increasing attentions, where a common assumption is that the class set for nodes is closed. However, in real-world scenarios, it often faces the open set problem with the dynamically increased class set as the time passes by. This will bring two big challenges to the existing dynamic GNN methods: (i) How to dynamically propagate appropriate information in an open temporal graph, where new class nodes are often linked to old class nodes. This case will lead to a sharp contradiction. This is because typical GNNs are prone to make the embeddings of connected nodes become similar, while we expect the embeddings of these two interactive nodes to be distinguishable since they belong to different classes. (ii) How to avoid catastrophic knowledge forgetting over old classes when learning new classes occurred in temporal graphs. In this paper, we propose a general and principled learning approach for open temporal graphs, called OTGNet, with the goal of addressing the above two challenges. We assume the knowledge of a node can be disentangled into class-relevant and class-agnostic one, and thus explore a new message passing mechanism by extending the information bottleneck principle to only propagate class-agnostic knowledge between nodes of different classes, avoiding aggregating conflictive information. Moreover, we devise a strategy to select both important and diverse triad sub-graph structures for effective class-incremental learning. Extensive experiments on three real-world datasets of different domains demonstrate the superiority of our method, compared to the baselines.

Motivation & Objective

• To address the open-set problem in temporal graph neural networks where new classes dynamically emerge over time.
• To resolve the heterophily propagation issue, where connected nodes from different classes are pushed toward similar embeddings by standard GNNs.
• To mitigate catastrophic forgetting during continual learning on evolving temporal graphs.
• To develop a principled message passing mechanism that only transfers class-agnostic knowledge between different-class nodes.
• To design an effective, theoretically grounded method for selecting representative triad subgraphs for knowledge rehearsal.

Proposed method

• Proposes a disentangled node representation learning framework, separating class-relevant and class-agnostic features.
• Introduces an information bottleneck-based message passing mechanism that selectively transmits only class-agnostic knowledge between nodes of different classes.
• Designs a value function to identify both important and diverse triad subgraphs for continual learning.
• Develops a greedy approximation algorithm with theoretical lower-bound guarantee for efficient triad selection.
• Integrates the selected triads into the training process as a form of experience replay to prevent forgetting of old classes.
• Employs a class-incremental learning strategy that updates the model incrementally upon arrival of new classes without retraining on all historical data.

Experimental results

Research questions

• RQ1 How can we design a message passing mechanism that avoids conflictive feature aggregation between nodes of different classes in open temporal graphs?
• RQ2 What criteria can effectively identify representative triad subgraphs that preserve both structural diversity and importance for continual learning?
• RQ3 Can a combination of disentangled representation learning and selective triad rehearsal significantly reduce catastrophic forgetting in temporal graph learning?
• RQ4 How does the proposed method compare to existing incremental learning baselines in terms of accuracy and forgetting rate on real-world temporal graph datasets?
• RQ5 What is the computational efficiency of the proposed triad selection algorithm, and how does it scale with increasing graph size?

Key findings

• OTGNet achieves the highest average precision (AP) on all three datasets: 73.88 on Reddit, 83.78 on Yelp, and 79.92 on Taobao, outperforming all baselines.
• On the Reddit dataset, OTGNet achieves 95.97 AP on the final task, significantly outperforming the second-best method (TGN+BiC at 80.15 AP).
• OTGNet reduces average forgetting (AF) to 12.82 on Taobao and 4.98 on Yelp, demonstrating strong stability across tasks.
• The t-SNE visualization shows that OTGNet learns more discriminative node embeddings than OTGNet-w.o.-IB, especially after new classes are introduced.
• The running time of OTGNet (6.78h on Reddit) is comparable to incremental baselines (5.05–6.23h) but significantly lower than retraining methods (10–35h).
• The proposed triad selection algorithm achieves a theoretical approximation ratio lower bound, ensuring a balance between diversity and importance in selected subgraphs.