[Paper Review] Graph-Bert: Only Attention is Needed for Learning Graph Representations

Jiawei Zhang, Haopeng Zhang+2 more|arXiv (Cornell University)|Jan 15, 2020

Advanced Graph Neural NetworksComputer Science28 references154 citations

TL;DR

Graph-Bert is a graph neural network that uses solely attention, trained on linkless subgraphs, with unsupervised pre-training and fine-tuning for tasks like node classification and graph clustering, achieving strong results on benchmark datasets.

ABSTRACT

The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes critical for large-sized graph, as memory constraints limit batching across the nodes. In this paper, we will introduce a new graph neural network, namely GRAPH-BERT (Graph based BERT), solely based on the attention mechanism without any graph convolution or aggregation operators. Instead of feeding GRAPH-BERT with the complete large input graph, we propose to train GRAPH-BERT with sampled linkless subgraphs within their local contexts. GRAPH-BERT can be learned effectively in a standalone mode. Meanwhile, a pre-trained GRAPH-BERT can also be transferred to other application tasks directly or with necessary fine-tuning if any supervised label information or certain application oriented objective is available. We have tested the effectiveness of GRAPH-BERT on several graph benchmark datasets. Based the pre-trained GRAPH-BERT with the node attribute reconstruction and structure recovery tasks, we further fine-tune GRAPH-BERT on node classification and graph clustering tasks specifically. The experimental results have demonstrated that GRAPH-BERT can out-perform the existing GNNs in both the learning effectiveness and efficiency.

Motivation & Objective

• Address inefficiencies and problems in traditional GNNs that rely on graph links (e.g., suspended animation and over-smoothing).
• Propose Graph-Bert, a transformer-based GNN that operates without graph convolutions and uses sampled linkless subgraphs for training.
• Enable unsupervised pre-training via node attribute reconstruction and graph structure recovery, followed by task-specific fine-tuning and transfer.
• Demonstrate that pre-trained Graph-Bert can transfer to related tasks and sequential models, improving learning efficiency and effectiveness.

Proposed method

• Introduce Graph-Bert architecture with five components: linkless subgraph batching, node input embeddings, graph-transformer based encoder, representation fusion, and a task-specific functional component.
• Use sampled subgraphs defined via a top-k intimacy sampling with an intimacy matrix based on PageRank to form learning contexts.
• Compute node input vectors by aggregating raw features, WL absolute role embeddings, intimacy-based relative positional embeddings, and hop-based relative distance embeddings.
• Process these embeddings through a graph-transformer encoder with multi-layer attention, plus a graph residual term that propagates information within each subgraph.
• Pre-train on two tasks—node attribute reconstruction and graph structure recovery—then fine-tune for node classification or graph clustering.

Experimental results

Research questions

• RQ1 Can Graph-Bert learn meaningful graph representations without relying on graph edge connectivity?
• RQ2 Do linkless subgraph batches plus attention-based encoding achieve competitive or superior performance compared to edge-based GNNs on standard benchmarks?
• RQ3 How effective are unsupervised pre-training tasks (node attribute reconstruction and structure recovery) for downstream graph tasks?
• RQ4 To what extent can Graph-Bert transfer to related tasks or sequential models without full re-training?

Key findings

• Graph-Bert outperforms several baseline methods on node classification across Cora, Citeseer, and Pubmed datasets (best reported 0.843 on Cora).
• Node classification performance on Citeseer and Pubmed is competitive, with Graph-Bert achieving 0.712 and 0.793 respectively in the reported results.
• Pre-training on node attribute reconstruction and structure recovery converges quickly and supports effective fine-tuning for downstream tasks.
• Graph-Bert supports very deep architectures without suffering from suspended animation, as shown by successful training up to 50 layers.
• Subgraph size k substantially impacts performance; there is a general trend of improved accuracy as k grows up to a point (e.g., Cora results shown up to k=7).
• With pre-training and fine-tuning, Graph-Bert achieves strong performance gains over multiple baselines, including GCN and GAT variants.