[Paper Review] Self-Supervised Representation Learning via Latent Graph Prediction

Yaochen Xie, Xu Zhao+1 more|arXiv (Cornell University)|Feb 16, 2022

Advanced Graph Neural Networks8 citations

TL;DR

This paper proposes LaGraph, a self-supervised representation learning framework for graph neural networks based on latent graph prediction. By deriving self-supervised upper bounds to predict unobserved latent graphs, LaGraph achieves state-of-the-art performance with improved robustness to small batch sizes and training on subgraphs, outperforming contrastive and predictive SSL methods on both node-level and graph-level benchmarks.

ABSTRACT

Self-supervised learning (SSL) of graph neural networks is emerging as a promising way of leveraging unlabeled data. Currently, most methods are based on contrastive learning adapted from the image domain, which requires view generation and a sufficient number of negative samples. In contrast, existing predictive models do not require negative sampling, but lack theoretical guidance on the design of pretext training tasks. In this work, we propose the LaGraph, a theoretically grounded predictive SSL framework based on latent graph prediction. Learning objectives of LaGraph are derived as self-supervised upper bounds to objectives for predicting unobserved latent graphs. In addition to its improved performance, LaGraph provides explanations for recent successes of predictive models that include invariance-based objectives. We provide theoretical analysis comparing LaGraph to related methods in different domains. Our experimental results demonstrate the superiority of LaGraph in performance and the robustness to decreasing of training sample size on both graph-level and node-level tasks.

Motivation & Objective

• To address the limitations of contrastive self-supervised learning (SSL) in graph neural networks, particularly the need for large batch sizes and negative samples.
• To develop a predictive SSL framework grounded in theoretical principles rather than heuristic design, unlike existing predictive models.
• To provide a theoretically justified alternative to contrastive methods that avoids collapse issues and reduces dependency on engineering tricks like stop gradient or EMA.
• To improve robustness in low-data regimes and large-scale graph settings by leveraging latent graph prediction as a pretext task.
• To explain the success of invariance-based methods like BGRL through a unified theoretical framework grounded in latent graph prediction.

Proposed method

• Formalizes the concept of a latent graph as a hidden, unobserved graph structure derived from node features and adjacency relations.
• Derives self-supervised upper bounds of the supervised objective for predicting the latent graph, enabling end-to-end training without labeled data.
• Introduces a dual-objective learning mechanism combining reconstruction of the latent graph and invariance regularization via data augmentation.
• Employs node masking as a data augmentation strategy to generate views, not for contrastive learning, but to enhance invariance in the latent prediction objective.
• Uses a graph encoder (e.g., GCN, GAT) to produce node and graph-level representations, with a readout function to aggregate graph-level embeddings.
• The training objective is derived from information-theoretic principles, linking it to mutual information maximization and providing theoretical grounding.

Experimental results

Research questions

• RQ1 Can a predictive self-supervised learning framework for GNNs be theoretically grounded in latent graph prediction, avoiding reliance on contrastive objectives and negative sampling?
• RQ2 How does the proposed latent graph prediction objective compare to contrastive and invariance-based methods in terms of performance and robustness?
• RQ3 To what extent does the framework maintain performance when training on small subsets of nodes or with reduced batch sizes?
• RQ4 Can the theoretical analysis of the latent graph prediction objective explain the success of existing invariance-based methods like BGRL?
• RQ5 Does the framework generalize across both node-level and graph-level representation learning tasks with consistent improvements?

Key findings

• LaGraph achieves state-of-the-art performance on 10 out of 12 node-level and graph-level benchmark datasets, including SOTA results on PPI, Reddit, and COLLAB.
• On the PPI dataset, LaGraph achieves a 74.6 F1-score, outperforming BGRL (70.5) and GraphCL (45.5), with significantly lower memory usage when training on small subsets.
• LaGraph maintains stable performance even when training on only 0.22% of nodes (100 nodes) on Flickr, achieving 6.07 F1-score, while GraphCL collapses at 1,000 nodes with 45.48 F1-score.
• LaGraph is robust to small batch sizes: performance remains stable when batch size is reduced from 128 to 8, unlike contrastive methods like GraphCL that suffer significant accuracy drops.
• LaGraph achieves 95.8% accuracy on COLLAB and 95.2% on Reddit, outperforming BGRL (95.6% and 95.2%) and other SOTA methods like GRACE and GCA.
• The ablation study confirms that LaGraph’s performance is stable even when training on subgraphs, with memory consumption growing linearly and performance remaining consistent, unlike GraphCL which becomes infeasible at scale.