[Paper Review] GPT-GNN: Generative Pre-Training of Graph Neural Networks

Ziniu Hu, Yuxiao Dong+3 more|arXiv (Cornell University)|Jun 27, 2020

Advanced Graph Neural Networks46 references71 citations

TL;DR

GPT-GNN pre-trains GNNs on unlabeled, large-scale graphs by jointly modeling attributed graph generation (attributes) and edge generation (structure), then fine-tunes on downstream tasks with few labels, achieving notable gains on Open Academic Graph and Amazon data.

ABSTRACT

Graph neural networks (GNNs) have been demonstrated to be powerful in modeling graph-structured data. However, training GNNs usually requires abundant task-specific labeled data, which is often arduously expensive to obtain. One effective way to reduce the labeling effort is to pre-train an expressive GNN model on unlabeled data with self-supervision and then transfer the learned model to downstream tasks with only a few labels. In this paper, we present the GPT-GNN framework to initialize GNNs by generative pre-training. GPT-GNN introduces a self-supervised attributed graph generation task to pre-train a GNN so that it can capture the structural and semantic properties of the graph. We factorize the likelihood of the graph generation into two components: 1) Attribute Generation and 2) Edge Generation. By modeling both components, GPT-GNN captures the inherent dependency between node attributes and graph structure during the generative process. Comprehensive experiments on the billion-scale Open Academic Graph and Amazon recommendation data demonstrate that GPT-GNN significantly outperforms state-of-the-art GNN models without pre-training by up to 9.1% across various downstream tasks.

Motivation & Objective

• Reduce labeling needs for downstream GNN tasks by leveraging self-supervised pre-training on large unlabeled graphs.
• Model both node attributes and graph structure to capture intrinsic graph semantics and dependencies.
• Provide an efficient pre-training framework that scales to billion-scale graphs and heterogeneous graphs.
• Demonstrate transferability of pre-trained GNNs across time and field settings on real-world datasets.

Proposed method

• Propose an attributed graph generation task that factorizes the likelihood of a graph into Attribute Generation and Edge Generation.
• Use a dependency-aware factorization to capture the interaction between node attributes and edges.
• Split nodes into Attribute Generation Nodes and Edge Generation Nodes to avoid information leakage during training.
• Employ an autoregressive graph generation objective over node permutations and use a decoupled decoding scheme for attributes and edges.
• Optimize attribute generation loss via a decoder predicting masked attributes and edge generation loss via a contrastive decoder over candidate neighbors.
• Utilize an adaptive embedding queue to provide large negative samples for edge-contrastive learning and enable subgraph-based training for large graphs.
• Extend the framework to heterogeneous and very large graphs using subgraph sampling (LADIES/HGSampling) and an Adaptive Queue to preserve global signal.

Experimental results

Research questions

• RQ1 Can a self-supervised, generative pre-training objective for graphs improve downstream GNN performance with limited labels?
• RQ2 How can we effectively model the coupling between node attributes and graph structure during pre-training?
• RQ3 Is GPT-GNN scalable to billion-scale graphs and applicable to heterogeneous graphs?
• RQ4 Do time-transfer and field-transfer settings support robust transfer of learned representations to unseen data?
• RQ5 What is the contribution of attribute vs. edge generation components to downstream gains?

Key findings

• GPT-GNN can boost downstream node classification and link prediction on Open Academic Graph (OAG) and Amazon data by up to 9.1% on average over state-of-the-art non-pre-trained models.
• The framework consistently improves performance when used with different base GNNs and across various transfer settings.
• Attribute generation and edge generation components each contribute to performance, with their combination yielding the best results.
• The method scales to billion-node graphs using subgraph sampling and adaptive negative sampling via a queue, enabling processing of large heterogeneous graphs.
• Pre-training on large graphs (OAG ~179M nodes, 2B edges; Amazon ~113M nodes) enables effective time-transfer and field-transfer to downstream tasks with limited labels.