[Paper Review] DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability

Weilin Cong, Yanhong Wu+5 more|arXiv (Cornell University)|Nov 19, 2021

Advanced Graph Neural Networks40 references4 citations

TL;DR

DyFormer is a scalable dynamic graph Transformer that leverages spatial-temporal encoding and a two-tower architecture with subgraph sampling to model implicit edges and temporal evolution, achieving consistent 1%–3% AUC gains over state-of-the-art methods on real-world dynamic graph benchmarks while improving generalization through provably beneficial self-supervised pre-training.

ABSTRACT

Transformers have achieved great success in several domains, including Natural Language Processing and Computer Vision. However, its application to real-world graphs is less explored, mainly due to its high computation cost and its poor generalizability caused by the lack of enough training data in the graph domain. To fill in this gap, we propose a scalable Transformer-like dynamic graph learning method named Dynamic Graph Transformer (DyFormer) with spatial-temporal encoding to effectively learn graph topology and capture implicit links. To achieve efficient and scalable training, we propose temporal-union graph structure and its associated subgraph-based node sampling strategy. To improve the generalization ability, we introduce two complementary self-supervised pre-training tasks and show that jointly optimizing the two pre-training tasks results in a smaller Bayesian error rate via an information-theoretic analysis. Extensive experiments on the real-world datasets illustrate that DyFormer achieves a consistent 1%-3% AUC gain (averaged over all time steps) compared with baselines on all benchmarks.

Motivation & Objective

• To address the poor generalization of graph neural networks on dynamic graphs due to missing or spurious links.
• To overcome scalability limitations in dynamic graph learning caused by increasing graph size and time steps.
• To improve model robustness and generalization through complementary self-supervised pre-training tasks grounded in information theory.
• To design an efficient training framework independent of graph size and temporal complexity using temporal-union graphs and targeted node sampling.
• To demonstrate state-of-the-art performance on dynamic link prediction across multiple real-world datasets.

Proposed method

• DyFormer employs a two-tower Transformer architecture with spatial-temporal encoding to inject both structural and temporal evolutionary inductive biases into the model.
• It introduces a temporal-union graph structure that aggregates information across multiple time steps into a unified meta-graph for efficient representation learning.
• A novel subgraph-based node sampling strategy enables training with complexity independent of graph size and number of time steps.
• The model uses full-attention mechanisms to capture implicit node relations beyond the input graph topology, enhancing robustness to missing or spurious links.
• Two complementary self-supervised pre-training tasks are designed to improve generalization, with theoretical justification via information-theoretic analysis showing reduced Bayesian error rate.
• The framework jointly optimizes pre-training objectives to improve downstream performance while maintaining scalability.

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture with spatial-temporal encoding effectively model dynamic graph evolution and capture implicit links?
• RQ2 Does joint optimization of two complementary self-supervised pre-training tasks lead to provably better generalization, as supported by information-theoretic analysis?
• RQ3 Can a subgraph-based sampling strategy with a temporal-union graph structure achieve training complexity independent of graph size and time steps?
• RQ4 How does DyFormer perform on dynamic link prediction compared to state-of-the-art baselines across real-world datasets?
• RQ5 How robust is DyFormer to noisy input graphs with missing or spurious links?

Key findings

• DyFormer achieves a consistent 1%–3% Macro AUC gain over all baselines across all real-world dynamic graph datasets on link prediction tasks.
• The model shows improved stability, with lower standard deviation in AUC scores across different random seeds, indicating more robust training.
• Pre-training significantly improves performance: DyFormer without pre-training shows lower AUC and higher variance, especially under high parameter complexity.
• On noisy graphs with 10%–50% artificially introduced spurious or missing links, DyFormer with full-attention aggregation maintains higher and more stable AUC scores than 1-hop attention variants.
• The full-attention variant achieves 86.12±0.57 Macro AUC on UCI and 71.64±0.18 on Yelp at 10% noise, outperforming 1-hop attention by over 3% in some settings.
• Ablation studies confirm the effectiveness of spatial-temporal encoding, the two-tower architecture, and the subgraph sampling strategy in enhancing performance and scalability.