[Paper Review] Attending to Graph Transformers

Luis T. Díaz Müller, Mikhail Galkin+2 more|arXiv (Cornell University)|Feb 8, 2023

Machine Learning in Materials ScienceMaterials Science32 citations

TL;DR

This paper provides a taxonomy and empirical evaluation of graph transformer architectures, emphasizing structural/positional encodings, input features, tokenization, and propagation strategies, and assesses their ability to recover graph structure and handle heterophily.

ABSTRACT

Recently, transformer architectures for graphs emerged as an alternative to established techniques for machine learning with graphs, such as (message-passing) graph neural networks. So far, they have shown promising empirical results, e.g., on molecular prediction datasets, often attributed to their ability to circumvent graph neural networks' shortcomings, such as over-smoothing and over-squashing. Here, we derive a taxonomy of graph transformer architectures, bringing some order to this emerging field. We overview their theoretical properties, survey structural and positional encodings, and discuss extensions for important graph classes, e.g., 3D molecular graphs. Empirically, we probe how well graph transformers can recover various graph properties, how well they can deal with heterophilic graphs, and to what extent they prevent over-squashing. Further, we outline open challenges and research direction to stimulate future work. Our code is available at https://github.com/luis-mueller/probing-graph-transformers.

Motivation & Objective

• Provide a structured overview of state-of-the-art graph transformer (GT) architectures and their theoretical properties.
• Survey structural and positional encodings and their impact on GT expressivity.
• Examine how GTs handle non-geometric vs geometric (3D) graph features.
• Evaluate GTs on structural awareness, heterophily, and over-squashing phenomena.
• Outline open challenges and future research directions in graph transformers.

Proposed method

• Derive a taxonomy of GT architectures across encodings, input features, tokenization, and propagation.
• Discuss theoretical expressivity of GTs and dependence on structural/positional biases.
• Survey and categorize structural and positional encodings (local/global/relative).
• Analyze input feature regimes (non-geometric vs geometric) and geometric equivariance approaches.
• Examine tokenization strategies (nodes, edges, subgraphs) and their computational trade-offs.
• Review propagation schemes (global, sparse, hybrid) and scaling techniques (linear/ kernelized attention).

Figure 2: Average train accuracy over ten random seeds of a GT on the NeighborsMatch dataset, compared to models from Alon & Yahav ( 2021 ) .

Experimental results

Research questions

• RQ1 How do structural and positional encodings affect GT expressivity and ability to distinguish non-isomorphic graphs?
• RQ2 Can GTs mitigate over-smoothing and over-squashing, especially on heterophilic graphs?
• RQ3 What are the practical trade-offs of node-only vs edge-inclusive vs subgraph tokenization for GTs?
• RQ4 How do geometric features (3D coordinates) and SE(3)/E(3) equivariance influence GT performance?
• RQ5 What are the effective propagation strategies to scale GTs to larger graphs while preserving accuracy?

Key findings

• GTs without expressive encodings have limited graph-structure discernment, reinforcing the need for structural/positional biases.
• Several encodings (RWSE, LapPE) and attention biases (shortest-path, edge representations) enhance structural awareness and task performance.
• Hybrid propagation (local GNN + global attention) often yields strong results, balancing scalability and expressivity.
• Geometric GTs with SE(3)/E(3) equivariance and distance-based attention improve molecular property tasks and 3D graph modeling.
• Empirical results show varying performance across tasks, with baselines (vanilla transformers) lagging behind structurally biased GTs on structural/heterophily benchmarks.
• Table 3 (heterophilic datasets) demonstrates substantial performance gaps between GNN baselines and GT variants with structural encodings.