[Paper Review] Bootstrapped Representation Learning on Graphs

Shantanu Thakoor, Corentin Tallec+4 more|arXiv (Cornell University)|Feb 12, 2021

Advanced Graph Neural NetworksComputer Science44 references54 citations

TL;DR

This paper proposes BGRL, a self-supervised graph representation learning method inspired by BYOL that eliminates the need for negative pairs, avoiding the quadratic computational and memory cost of contrastive learning. By using a bootstrap mechanism with target and online networks, BGRL achieves state-of-the-art performance, including 70.49% Micro-F1 on PPI using GAT, outperforming prior unsupervised methods and matching or exceeding supervised GNNs on other benchmarks.

ABSTRACT

Current state-of-the-art self-supervised learning methods for graph neural networks (GNNs) are based on contrastive learning. As such, they heavily depend on the construction of augmentations and negative examples. For example, on the standard PPI benchmark, increasing the number of negative pairs improves performance, thereby requiring computation and memory cost quadratic in the number of nodes to achieve peak performance. Inspired by BYOL, a recently introduced method for self-supervised learning that does not require negative pairs, we present Bootstrapped Graph Latents, BGRL, a self-supervised graph representation method that gets rid of this potentially quadratic bottleneck. BGRL outperforms or matches the previous unsupervised state-of-the-art results on several established benchmark datasets. Moreover, it enables the effective usage of graph attentional (GAT) encoders, allowing us to further improve the state of the art. In particular on the PPI dataset, using GAT as an encoder we achieve state-of-the-art 70.49% Micro-F1, using the linear evaluation protocol. On all other datasets under consideration, our model is competitive with the equivalent supervised GNN results, often exceeding them.

Motivation & Objective

• To address the high computational and memory cost of contrastive self-supervised learning in GNNs, which scales quadratically with the number of nodes due to negative pair construction.
• To develop a self-supervised graph representation learning method that does not rely on negative examples, inspired by BYOL's success in vision.
• To enable effective use of graph attention mechanisms (GAT) in unsupervised pre-training for improved performance.
• To achieve state-of-the-art performance on standard graph benchmark datasets under linear evaluation protocols.
• To demonstrate that self-supervised GNNs can match or exceed supervised GNN performance on multiple datasets.

Proposed method

• Proposes BGRL, a bootstrap-based self-supervised learning framework for graphs, using two neural networks: an online network and a target network.
• The target network is a slow-moving exponential moving average (EMA) of the online network's weights, providing a target for contrastive-like learning without negative pairs.
• The model uses data augmentations (e.g., node masking, edge dropping) to create two views of the same graph, which are encoded by the online and target networks.
• Contrastive learning is avoided by minimizing the distance between the representations of the two augmented views, without requiring negative samples.
• The method is trained end-to-end using a stop-gradient operation on the target network's features, stabilizing learning without negative pairs.
• The framework is compatible with various GNN architectures, including GAT, enabling improved performance through attention mechanisms.

Experimental results

Research questions

• RQ1 Can a self-supervised graph learning method eliminate the need for negative pairs without sacrificing performance?
• RQ2 Can a bootstrap-based approach like BYOL be successfully adapted to graph-structured data to avoid quadratic complexity in negative pair computation?
• RQ3 Does BGRL enable better performance when combined with graph attention mechanisms (GAT) compared to previous unsupervised methods?
• RQ4 Can unsupervised pre-training with BGRL achieve performance competitive with or superior to supervised GNNs on standard benchmarks?
• RQ5 How does BGRL scale in terms of computational cost compared to contrastive methods, especially on large graphs?

Key findings

• BGRL achieves a state-of-the-art 70.49% Micro-F1 on the PPI dataset using the linear evaluation protocol, outperforming previous unsupervised methods.
• On all other benchmark datasets evaluated, BGRL's performance is competitive with or exceeds that of supervised GNNs.
• The method significantly reduces computational and memory costs by eliminating the need for negative pairs, avoiding quadratic scaling with the number of nodes.
• BGRL enables effective use of GAT encoders in unsupervised pre-training, leading to improved performance on PPI.
• The model maintains strong performance across multiple datasets, demonstrating generalization and robustness to different graph structures.
• The ablation study confirms that the bootstrap mechanism with EMA and stop-gradient is essential for stable training and high performance.