[Paper Review] Automated Self-Supervised Learning for Graphs

Wei Jin, Xiaorui Liu+4 more|arXiv (Cornell University)|Jun 10, 2021

Advanced Graph Neural NetworksComputer Science59 references21 citations

TL;DR

This paper proposes AutoSSL, an automated framework that searches for optimal combinations of self-supervised learning (SSL) tasks in graphs using a pseudo-homophily measure as an unsupervised surrogate for downstream performance. By leveraging the homophily principle—where connected nodes tend to share labels—the method dynamically weights multiple SSL tasks (e.g., DGI, Pairsim, PairDis) to improve node representation learning, achieving state-of-the-art performance on node clustering and classification across 8 real-world datasets.

ABSTRACT

Graph self-supervised learning has gained increasing attention due to its capacity to learn expressive node representations. Many pretext tasks, or loss functions have been designed from distinct perspectives. However, we observe that different pretext tasks affect downstream tasks differently cross datasets, which suggests that searching pretext tasks is crucial for graph self-supervised learning. Different from existing works focusing on designing single pretext tasks, this work aims to investigate how to automatically leverage multiple pretext tasks effectively. Nevertheless, evaluating representations derived from multiple pretext tasks without direct access to ground truth labels makes this problem challenging. To address this obstacle, we make use of a key principle of many real-world graphs, i.e., homophily, or the principle that "like attracts like," as the guidance to effectively search various self-supervised pretext tasks. We provide theoretical understanding and empirical evidence to justify the flexibility of homophily in this search task. Then we propose the AutoSSL framework which can automatically search over combinations of various self-supervised tasks. By evaluating the framework on 7 real-world datasets, our experimental results show that AutoSSL can significantly boost the performance on downstream tasks including node clustering and node classification compared with training under individual tasks. Code is released at https://github.com/ChandlerBang/AutoSSL.

Motivation & Objective

• To address the challenge of selecting optimal self-supervised learning (SSL) tasks for graph representation learning, as individual tasks perform variably across datasets and downstream tasks.
• To overcome the lack of ground-truth labels during training by introducing a pseudo-homophily measure that serves as an unsupervised proxy for downstream performance.
• To develop an automated framework that efficiently searches over combinations of multiple SSL tasks to jointly optimize representation quality.
• To empirically validate that higher pseudo-homophily correlates with better downstream performance, justifying its use as a search objective.

Proposed method

• Proposes a pseudo-homophily measure to estimate the alignment between predicted node labels (from embeddings) and true labels without access to ground truth, based on the homophily principle.
• Uses the pseudo-homophily score as a surrogate objective to guide the search for optimal task weights in multi-task SSL, with theoretical justification that maximizing it improves the upper bound of mutual information with ground-truth labels.
• Develops two search algorithms: AutoSSL-es (evolutionary strategy) and AutoSSL-ds (differentiable search) to efficiently explore the high-dimensional space of SSL task combinations.
• Employs a differentiable or gradient-based optimization process in AutoSSL-ds and a population-based evolutionary strategy in AutoSSL-es to iteratively refine task weights.
• Integrates multiple SSL tasks including DGI, Pairsim, PairDis, Clu, and Par, each capturing distinct inductive biases (e.g., local vs. global structure).
• Trains GNNs using weighted combinations of SSL objectives and evaluates the resulting embeddings via downstream node clustering and classification.

Experimental results

Research questions

• RQ1 How do different self-supervised learning tasks perform across various graph datasets and downstream tasks?
• RQ2 Can pseudo-homophily serve as a reliable unsupervised proxy for evaluating the quality of learned node representations in the absence of labels?
• RQ3 Can an automated framework effectively search over multiple SSL tasks to improve downstream performance compared to using individual tasks?
• RQ4 How do the optimal weights of different SSL tasks vary across different types of graphs (e.g., citation vs. co-purchase networks)?

Key findings

• AutoSSL significantly improves downstream performance on node clustering and classification across 8 real-world datasets compared to training with individual SSL tasks.
• The pseudo-homophily measure strongly correlates with downstream performance: embeddings with higher pseudo-homophily achieve better clustering (NMI) and classification (ACC) accuracy.
• On citation and co-authorship networks (e.g., CoraFull, Physics, CS), DGI and Par are consistently assigned high weights, indicating their importance in capturing global and local structural patterns.
• On co-purchase networks (e.g., Photo, Computers), Par and DGI are both important, but Par receives higher weight, suggesting local structure is more critical in these graphs.
• Clu consistently receives very low weights across datasets, indicating that pseudo-labels from raw features are poor supervision signals in the selected settings.
• The evolution of task weights in AutoSSL shows convergence to stable, dataset-specific configurations, with both AutoSSL-es and AutoSSL-ds achieving high pseudo-homophily and downstream accuracy.