[Paper Review] GraphiT: Encoding Graph Structure in Transformers

Grégoire Mialon, Dexiong Chen+2 more|arXiv (Cornell University)|Jun 10, 2021

Advanced Graph Neural NetworksComputer Science29 references42 citations

TL;DR

GraphiT shows that a vanilla transformer can outperform GNNs on graph tasks when augmented with kernel-based relative positional encodings and local substructure (GCKN) features, enabling competitive or superior results across benchmarks and offering interpretable attention-driven motifs.

ABSTRACT

We show that viewing graphs as sets of node features and incorporating structural and positional information into a transformer architecture is able to outperform representations learned with classical graph neural networks (GNNs). Our model, GraphiT, encodes such information by (i) leveraging relative positional encoding strategies in self-attention scores based on positive definite kernels on graphs, and (ii) enumerating and encoding local sub-structures such as paths of short length. We thoroughly evaluate these two ideas on many classification and regression tasks, demonstrating the effectiveness of each of them independently, as well as their combination. In addition to performing well on standard benchmarks, our model also admits natural visualization mechanisms for interpreting graph motifs explaining the predictions, making it a potentially strong candidate for scientific applications where interpretation is important. Code available at https://github.com/inria-thoth/GraphiT.

Motivation & Objective

• Motivate using transformer architectures for graph-structured data as an alternative to traditional GNNs.
• Develop methods to inject graph structure into transformers through position encoding and local substructure features.
• Evaluate the impact of kernel-based relative positional encodings and GCKN-derived features across classification and regression benchmarks.
• Provide visualization techniques to interpret attention and identify meaningful graph motifs.

Proposed method

• Process graphs with a vanilla transformer encoder acting on node features.
• Introduce relative positional encoding in attention using positive definite kernels on graphs (e.g., diffusion, random-walk kernels).
• Enrich node features with local substructure representations via Graph Convolutional Kernel Networks (GCKN) paths features.
• Combine relative positional encoding and structure-aware node features, and compare with Laplacian-based LapPE and GCKN baselines.
• Experiment with global vs. local attention (full communication vs. neighborhood-like aggregation) and ablations across datasets.
• Provide attention-based visualization to identify discriminative graph motifs and substructures.

Experimental results

Research questions

• RQ1 Can vanilla transformers, with proper graph-aware position encodings, outperform GNNs on graph classification and regression tasks?
• RQ2 Are kernel-based relative positional encodings superior to Laplacian-based absolute position encodings (LapPE) for graphs?
• RQ3 What combination of structure encoding and relative positional encoding yields the best performance across datasets?
• RQ4 Do attention scores reveal meaningful graph motifs that align with known substructures in chemistry or biology?

Key findings

• GraphiT variants achieve comparable or superior performance to common GNNs on MUTAG, PROTEINS, PTC, and NCI1 datasets.
• On the ZINC regression task, GraphiT with structure and relative position encoding outperforms all prior baselines by a large margin.
• GCKN-path features frequently outperform LapPE across several datasets, particularly when combined with adjacency-based or RW-based relative encodings.
• Combining structure encoding in node features (e.g., GCKN-path) with relative positional encoding yields the best overall results on several benchmarks, including ZINC.
• Relative kernel-based position encoding generally outperforms LapPE, and the combination of both yields substantial gains.
• Attention visualizations reveal chemically meaningful substructures (e.g., nitro groups) contributing to mutagenicity predictions.