[论文解读] TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning

Lu Wang, Xiaofu Chang+8 more|arXiv (Cornell University)|May 17, 2021

Advanced Graph Neural Networks参考文献 32被引用 24

一句话总结

TCL introduces a graph-topology-aware transformer for continuous-time dynamic graphs, employs a two-stream encoder with co-attention for temporal neighborhoods, and uses a future-state contrastive objective to maximize mutual information between interacting nodes.

ABSTRACT

Dynamic graph modeling has recently attracted much attention due to its extensive applications in many real-world scenarios, such as recommendation systems, financial transactions, and social networks. Although many works have been proposed for dynamic graph modeling in recent years, effective and scalable models are yet to be developed. In this paper, we propose a novel graph neural network approach, called TCL, which deals with the dynamically-evolving graph in a continuous-time fashion and enables effective dynamic node representation learning that captures both the temporal and topology information. Technically, our model contains three novel aspects. First, we generalize the vanilla Transformer to temporal graph learning scenarios and design a graph-topology-aware transformer. Secondly, on top of the proposed graph transformer, we introduce a two-stream encoder that separately extracts representations from temporal neighborhoods associated with the two interaction nodes and then utilizes a co-attentional transformer to model inter-dependencies at a semantic level. Lastly, we are inspired by the recently developed contrastive learning and propose to optimize our model by maximizing mutual information (MI) between the predictive representations of two future interaction nodes. Benefiting from this, our dynamic representations can preserve high-level (or global) semantics about interactions and thus is robust to noisy interactions. To the best of our knowledge, this is the first attempt to apply contrastive learning to representation learning on dynamic graphs. We evaluate our model on four benchmark datasets for interaction prediction and experiment results demonstrate the superiority of our model.

研究动机与目标

• Motivate dynamic graph representation learning that captures both temporal evolution and topology.
• Propose a Transformer-based framework that handles irregular time intervals in TDIGs.
• Introduce a two-stream encoder to separately model temporal neighborhoods of interacting nodes and fuse them semantically.
• Incorporate a contrastive learning objective to maximize mutual information between future interaction node representations.

提出的方法

• Extend vanilla Transformer to graph-topology-aware processing with time-interval and depth embeddings.
• Inject TDIG structure into attention using a masked multi-head attention to respect temporal dependencies.
• Implement a two-stream encoder that processes each interaction node’s temporal neighborhood and then applies a co-attentional Transformer to fuse representations.
• Predict future node representations from dependency nodes and optimize via a contrastive loss that maximizes agreement between future node representations of interacting pairs.
• Use a Sim function combining additive and multiplicative terms to score positive vs. negative pairs in the contrastive objective.

实验结果

研究问题

• RQ1 Can a Transformer-based architecture effectively model temporal and topological information in continuous-time dynamic graphs (TDIG)?
• RQ2 Does a two-stream, co-attentional encoder improve dynamic embeddings by leveraging semantic relations between temporal neighborhoods?
• RQ3 Can contrastive learning on future interaction representations yield robust, noise-tolerant dynamic graph representations and improve interaction prediction?

主要发现

• A graph-topology-aware Transformer with masked attention can encode temporal-topological information in TDIGs.
• A two-stream encoder with a co-attentional Transformer yields more informative dynamic embeddings by modeling cross-neighborhood semantics.
• A contrastive learning objective based on future predictions improves robustness to noise and helps maximize mutual information between interacting nodes.
• The TCL framework shows consistent improvements over baselines on four dynamic interaction datasets (no numerical values provided in the source).