[Paper Review] Graph Contrastive Learning with Augmentations

Yuning You, Tianlong Chen+4 more|arXiv (Cornell University)|Oct 22, 2020

Advanced Graph Neural Networks71 references863 citations

TL;DR

GraphCL introduces graph-level data augmentations and a contrastive learning framework for GNN pre-training, achieving state-of-the-art results in semi-supervised, unsupervised, transfer learning, and adversarial robustness across multiple graph datasets.

ABSTRACT

Generalizable, transferrable, and robust representation learning on graph-structured data remains a challenge for current graph neural networks (GNNs). Unlike what has been developed for convolutional neural networks (CNNs) for image data, self-supervised learning and pre-training are less explored for GNNs. In this paper, we propose a graph contrastive learning (GraphCL) framework for learning unsupervised representations of graph data. We first design four types of graph augmentations to incorporate various priors. We then systematically study the impact of various combinations of graph augmentations on multiple datasets, in four different settings: semi-supervised, unsupervised, and transfer learning as well as adversarial attacks. The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, our GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods. We also investigate the impact of parameterized graph augmentation extents and patterns, and observe further performance gains in preliminary experiments. Our codes are available at https://github.com/Shen-Lab/GraphCL.

Motivation & Objective

• Motivate pre-training for graph neural networks to improve generalization and transfer across diverse graph types.
• Design four graph data augmentations to encode domain priors and enable robust contrastive learning.
• Propose GraphCL to learn invariant graph representations by maximizing agreement between augmented views.
• Systematically study augmentation effects across datasets and task settings to guide practical use.

Proposed method

• Propose four graph data augmentations: node dropping, edge perturbation, attribute masking, and subgraph sampling.
• Use a shared GNN encoder f and a projection head g to map augmented graphs to representations.
• Apply NT-Xent contrastive loss to maximize agreement between two augmented views of the same graph.
• Show GraphCL corresponds to a mutual information maximization objective between latent representations of augmented graphs.
• Demonstrate that GraphCL can unify a broad family of graph contrastive methods under a general framework.

Experimental results

Research questions

• RQ1 How do different graph augmentation types and their configurations affect contrastive learning for graphs?
• RQ2 Can a GraphCL framework achieve better generalization and transferability than existing pre-training methods for GNNs?
• RQ3 What is the impact of augmentation strength and pattern on downstream performance across graph domains (biochemical, social, image-based graphs)?
• RQ4 Does GraphCL improve robustness of GNNs against adversarial perturbations?

Key findings

• Graph augmentations are crucial; without them graph contrastive learning is not beneficial compared to training from scratch.
• Composing augmentations (different types) generally improves performance more than using a single type.
• Edge perturbation helps social networks but can hurt some biomolecule datasets, indicating domain-specific priors are important.
• Attribute masking and node dropping/subgraph augmentations yield consistent gains, with patterns (e.g., masking hubs in dense graphs) further improving results.
• GraphCL achieves state-of-the-art performance in semi-supervised, unsupervised representation learning, and transfer learning on multiple graph classification benchmarks.
• GraphCL also boosts robustness to adversarial attacks compared to training from scratch.