[Paper Review] GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph

Junhan Yang, Zheng Liu+7 more|arXiv (Cornell University)|May 6, 2021

Advanced Graph Neural Networks51 citations

TL;DR

GraphFormers fuses GNNs with transformer blocks in a nested architecture to jointly encode text and graph structure, using a two-stage progressive learning strategy, achieving state-of-the-art results on large textual graph benchmarks with comparable efficiency.

ABSTRACT

The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information. Recent breakthroughs on pretrained language models and graph neural networks push forward the development of corresponding techniques. The existing works mainly rely on the cascaded model architecture: the textual features of nodes are independently encoded by language models at first; the textual embeddings are aggregated by graph neural networks afterwards. However, the above architecture is limited due to the independent modeling of textual features. In this work, we propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow, {making} each node's semantic accurately comprehended from the global perspective. In addition, a {progressive} learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph. Extensive evaluations are conducted on three large-scale benchmark datasets, where GraphFormers outperform the SOTA baselines with comparable running efficiency.

Motivation & Objective

• Motivate accurate representations for textual graphs by jointly modeling node text and neighborhood structure.
• Propose a GNN-nested Transformer architecture that iteratively fuses text encoding and graph aggregation.
• Introduce a two-stage progressive learning regime to strengthen information integration from graph structure.
• Demonstrate that GraphFormers outperform state-of-the-art cascaded Transformer-GNN baselines under comparable efficiency.

Proposed method

• Nest GNN components alongside Transformer blocks to fuse text encoding and graph aggregation in an iterative workflow.
• Use layerwise GNN with multi-head attention to produce node-level messages aggregated across neighbors.
• Employ an asymmetric multi-head attention in the Transformer to allow center-to-neighbor references while enabling caching of neighbor encodings.
• Introduce graph-augmented token embeddings by concatenating neighbor-informed node embeddings with token embeddings at each layer.
• Train with a two-stage progressive learning objective: first on polluted inputs to force graph information usage, then on original data for target distribution.
• Optionally use a unidirectional graph aggregation to cache neighbor encodings and reduce computation.

Experimental results

Research questions

• RQ1 How can we integrate GNN-based neighborhood information directly into Transformer-based text encoders for textual graphs?
• RQ2 Does a nested GNN-Transformer architecture improve node representation quality compared with cascaded text-first then graph aggregation models?
• RQ3 Can a progressive, two-stage training strategy improve the model’s ability to leverage graph information while maintaining efficiency?
• RQ4 What is the impact of unidirectional graph aggregation on performance and computation cost?

Key findings

• GraphFormers consistently outperforms state-of-the-art cascaded Transformer-GNN baselines across three large-scale textual graph datasets (Product, DBLP, Wiki).
• Shows relative improvements of 2.9%, 4.8%, and 6.5% on the three datasets respectively over strongest baselines.
• Maintains comparable efficiency to cascaded methods, with GraphFormers’ time and memory costs remaining close to PLM+Max baselines even as neighborhood size grows.
• Two-stage progressive learning significantly improves representation quality by about 1%–2% in key metrics when compared to training directly on original data.
• A unidirectional graph aggregation variant preserves performance while substantial caching reduces repeated neighbor encodings.”
• Ablation studies show the necessity of progressive learning and that position biases and shared GNNs contribute modestly to performance.