[Paper Review] Self-supervised Auxiliary Learning for Graph Neural Networks via Meta-Learning

Dasol Hwang, Jinyoung Park+4 more|ArXiv.org|Mar 1, 2021

Advanced Graph Neural Networks73 references4 citations

TL;DR

This paper proposes SELAR, a meta-learning framework that automatically selects and balances self-supervised auxiliary tasks—particularly meta-path prediction—for heterogeneous graph neural networks to improve primary tasks like node classification and link prediction. By learning task-specific weighting functions and using a Hint Network to correct predictions, SELAR enhances representation learning without manual labeling or additional data, achieving consistent performance gains across datasets.

ABSTRACT

In recent years, graph neural networks (GNNs) have been widely adopted in the representation learning of graph-structured data and provided state-of-the-art performance in various applications such as link prediction, node classification, and recommendation. Motivated by recent advances of self-supervision for representation learning in natural language processing and computer vision, self-supervised learning has been recently studied to leverage unlabeled graph-structured data. However, employing self-supervision tasks as auxiliary tasks to assist a primary task has been less explored in the literature on graphs. In this paper, we propose a novel self-supervised auxiliary learning framework to effectively learn graph neural networks. Moreover, this work is the first study showing that a meta-path prediction is beneficial as a self-supervised auxiliary task for heterogeneous graphs. Our method is learning to learn a primary task with various auxiliary tasks to improve generalization performance. The proposed method identifies an effective combination of auxiliary tasks and automatically balances them to improve the primary task. Our methods can be applied to any graph neural network in a plug-in manner without manual labeling or additional data. Also, it can be extended to any other auxiliary tasks. Our experiments demonstrate that the proposed method consistently improves the performance of node classification and link prediction.

Motivation & Objective

• To address the challenge of negative transfer in multi-task GNN training by automatically identifying effective auxiliary tasks for graph-structured data.
• To improve generalization in graph neural networks without manual labeling or additional data by leveraging self-supervised learning on heterogeneous graphs.
• To develop a plug-in framework compatible with any GNN architecture that dynamically balances primary and auxiliary tasks via meta-learning.
• To introduce meta-path prediction as a novel, self-supervised auxiliary task specifically designed for heterogeneous graphs to capture rich semantic relationships.
• To enhance model performance on downstream tasks by learning a dynamic weighting function that focuses on hard examples and improves knowledge transfer.

Proposed method

• The framework formulates auxiliary task selection and loss weighting as a meta-learning problem, where a meta-learner optimizes the loss weights for multiple tasks.
• It generates meta-paths from heterogeneous graphs automatically, without manual labeling, to serve as self-supervised auxiliary tasks.
• A Hint Network is introduced to dynamically correct the learner’s predictions on challenging auxiliary tasks, improving knowledge transfer.
• The method uses a task-specific weighting function that adjusts the cross-entropy loss, similar to focal loss, to down-weight easy examples and focus on hard ones.
• The weighting function evolves over training, initially focusing on easy samples and later shifting to hard examples, with higher attention on the primary task.
• The framework is plug-and-play, compatible with any GNN architecture and extendable to both self-supervised and supervised auxiliary tasks.

Experimental results

Research questions

• RQ1 Can self-supervised auxiliary tasks improve the performance of graph neural networks on primary tasks like node classification and link prediction?
• RQ2 Is meta-path prediction an effective self-supervised auxiliary task for heterogeneous graphs, and does it provide better representation learning than other auxiliary tasks?
• RQ3 Can a meta-learning approach automatically identify and balance multiple auxiliary tasks to avoid negative transfer and improve generalization?
• RQ4 How does the dynamic weighting function affect learning, particularly in focusing on hard examples across tasks?
• RQ5 Can a Hint Network further enhance performance by correcting predictions on difficult auxiliary tasks?

Key findings

• SELAR consistently improves performance on node classification and link prediction across multiple datasets, including Last-FM and Book-Crossing.
• Meta-path prediction as an auxiliary task provides a significant performance gain, especially in capturing user preference patterns in heterogeneous graphs.
• The weighting function learned by SELAR focuses more on hard examples, particularly in the primary task, and dynamically shifts focus from easy to hard samples during training.
• The Hint Network significantly improves gains from auxiliary tasks, especially when they are challenging or remotely relevant to the primary task.
• On the Book-Crossing dataset, the framework assigns higher attention to the 'user-item' primary task and the 'user-item-literary.series.item-user' meta-path, indicating effective capture of user preference similarity.
• The weighting function converges to near-zero values for very low or high losses in later training, indicating reduced importance for already well-classified or misclassified samples.