[Paper Review] Do Transformers Really Perform Bad for Graph Representation?

Chengxuan Ying, Tianle Cai+6 more|arXiv (Cornell University)|Jun 9, 2021

Advanced Graph Neural Networks62 references126 citations

TL;DR

Graphormer demonstrates that plain Transformer architectures, with graph-specific structural encodings, achieve state-of-the-art results on major graph representation benchmarks, including OGB-LSC.

ABSTRACT

The Transformer architecture has become a dominant choice in many domains, such as natural language processing and computer vision. Yet, it has not achieved competitive performance on popular leaderboards of graph-level prediction compared to mainstream GNN variants. Therefore, it remains a mystery how Transformers could perform well for graph representation learning. In this paper, we solve this mystery by presenting Graphormer, which is built upon the standard Transformer architecture, and could attain excellent results on a broad range of graph representation learning tasks, especially on the recent OGB Large-Scale Challenge. Our key insight to utilizing Transformer in the graph is the necessity of effectively encoding the structural information of a graph into the model. To this end, we propose several simple yet effective structural encoding methods to help Graphormer better model graph-structured data. Besides, we mathematically characterize the expressive power of Graphormer and exhibit that with our ways of encoding the structural information of graphs, many popular GNN variants could be covered as the special cases of Graphormer.

Motivation & Objective

• Demonstrate that Transformers can be competitive for graph representation learning.
• Introduce structural encodings to inject graph structure into Transformer attention.
• Provide theoretical analysis showing Graphormer’s expressiveness encompasses common GNNs.
• Empirically validate on large-scale and standard graph benchmarks like OGB-LSC, MolPCBA, MolHIV, and ZINC.

Proposed method

• Propose Graphormer, a standard Transformer adapted for graphs with three structural encodings: Centrality Encoding, Spatial Encoding, and Edge Encoding.
• Centrality Encoding adds degree-based embeddings to node features at input to inform attention.
• Spatial Encoding biases attention via learned scalars b_{φ(v_i,v_j)} derived from graph-based node-pair relations (e.g., shortest-path distance).
• Edge Encoding incorporates edge features along shortest paths into attention via an aggregated bias c_{ij} across the path.
• Introduce a special [VNode] token connected to all nodes to support graph-level Readout, analogous to [CLS] in NLP models.
• Use Transformer encoder blocks with pre-LN configuration and a Readout via the [VNode].
• Provide theoretical results showing Graphormer can simulate GNN aggregation/combination and surpass 1-WL expressiveness under the proposed encodings.

Experimental results

Research questions

• RQ1 Can a vanilla Transformer, when augmented with graph-aware structural encodings, match or surpass traditional GNNs on graph-level prediction tasks?
• RQ2 How do centrality, spatial, and edge encodings contribute to Graphormer’s performance and representational power?
• RQ3 Is Graphormer expressive enough to subsume common GNN variants (e.g., GCN, GIN) as special cases?
• RQ4 What is Graphormer’s performance on large-scale graph benchmarks (e.g., OGB-LSC PCQM4M-LSC) and standard benchmarks (MolPCBA, MolHIV, ZINC) after ablations?

Key findings

• Graphormer with the proposed encodings achieves state-of-the-art or competitive results across large-scale and standard graph benchmarks.
• On PCQM4M-LSC, Graphormer (full) achieves a validate MAE of 0.1234, substantially better than many GNN baselines; Graphormer Small also performs strongly with 0.1264 validate MAE.
• Graphormer outperforms prior SOTA GNNs on MolPCBA (AP 31.39 ±0.32) and MolHIV (AUC 80.51 ±0.53) with FLAG augmentation.
• On ZINC, Graphormer-SLIM achieves 0.122 ±0.006 MAE, surpassing several traditional GNNs and Transformer-based competitors.
• Ablation studies show Spatial Encoding and Centrality Encoding significantly improve performance; edge encoding via attention bias yields further gains.
• The experiments indicate that many popular GNN variants can be recovered as special cases of Graphormer under appropriate encodings and weights.