[Paper Review] Temporal Graph Networks for Deep Learning on Dynamic Graphs

Emanuele Rossi, Ben Chamberlain+4 more|arXiv (Cornell University)|Jun 18, 2020

Advanced Graph Neural NetworksComputer Science82 references95 citations

TL;DR

Temporal Graph Networks (TGNs) provide a generic, memory-enhanced framework for learning on continuous-time dynamic graphs represented as sequences of time-stamped events, achieving state-of-the-art results with efficient training.

ABSTRACT

Graph Neural Networks (GNNs) have recently become increasingly popular due to their ability to learn complex systems of relations or interactions arising in a broad spectrum of problems ranging from biology and particle physics to social networks and recommendation systems. Despite the plethora of different models for deep learning on graphs, few approaches have been proposed thus far for dealing with graphs that present some sort of dynamic nature (e.g. evolving features or connectivity over time). In this paper, we present Temporal Graph Networks (TGNs), a generic, efficient framework for deep learning on dynamic graphs represented as sequences of timed events. Thanks to a novel combination of memory modules and graph-based operators, TGNs are able to significantly outperform previous approaches being at the same time more computationally efficient. We furthermore show that several previous models for learning on dynamic graphs can be cast as specific instances of our framework. We perform a detailed ablation study of different components of our framework and devise the best configuration that achieves state-of-the-art performance on several transductive and inductive prediction tasks for dynamic graphs.

Motivation & Objective

• Motivate learning on dynamic graphs where interactions occur over continuous time rather than in discrete snapshots.
• Introduce Temporal Graph Networks (TGNs) as a versatile encoder for continuous-time dynamic graphs.
• Enable memory-based per-node representations to capture long-term history in dynamic interactions.
• Develop an efficient training strategy that propagates gradients through memory-related components without leakage.
• Demonstrate state-of-the-art performance on transductive and inductive tasks while improving computational efficiency.

Proposed method

• Define a continuous-time dynamic graph as a sequence of time-stamped events (node-wise and interaction events).
• Introduce a memory module that stores a per-node state to capture history across events.
• Compute messages from events using message functions, then aggregate them, and update per-node memories with a memory updater.
• Provide an embedding module (options include identity, time projection, temporal graph attention, and temporal graph sum) to generate up-to-date node embeddings from memory and neighbor information.
• Implement an embedding strategy that mitigates memory staleness by incorporating neighborhood information up to a chosen hop count with time-aware encodings.
• Adopt a training scheme that stores raw messages to avoid information leakage, enabling gradient flow through memory-related modules while processing batches.
• Evaluate TGNs on edge prediction (transductive and inductive) and dynamic node classification across Wikipedia, Reddit, and Twitter datasets, comparing to CTDNE, Jodie, DyRep, TGAT, and static baselines.

Experimental results

Research questions

• RQ1 Can a memory-augmented, event-driven encoder outperform existing continuous-time dynamic graph models on both transductive and inductive tasks?
• RQ2 Does incorporating graph-based embedding and temporal neighborhood information yield more up-to-date node representations and improve predictive performance?
• RQ3 How does memory design (presence/absence, aggregation strategy) affect accuracy and training efficiency?
• RQ4 What trade-offs exist between model complexity (layers, attention mechanisms) and speed when learning on dynamic graphs?

Key findings

• TGNs achieve state-of-the-art results on multiple dynamic graph tasks and datasets for both edge prediction and node classification.
• Models with memory outperform memoryless variants, demonstrating the importance of storing long-term node history.
• Graph-based embedding modules (especially temporal graph attention) significantly boost performance over memory-only or graph-less variants.
• Memory-enabled TGNs are substantially faster (up to 30x per epoch) than TGAT while maintaining competitive or superior accuracy.
• A single graph attention layer with memory suffices for strong performance, due to memory access providing broader temporal context.