[論文レビュー] Self-supervised Learning on Graphs: Deep Insights and New Direction

Wei Jin, Tyler Derr+5 more|arXiv (Cornell University)|Jun 17, 2020

Advanced Graph Neural Networks参考文献 32被引用数 111

ひとこと要約

本論文は、さまざまな自己教師あり学習タスクがノード分類のためのGNNにどのような影響を与えるかを分析し、結合訓練と二段階訓練を比較し、高度な前テキストタスク(SelfTask)へと繋がる洞察を提案します。

ABSTRACT

The success of deep learning notoriously requires larger amounts of costly annotated data. This has led to the development of self-supervised learning (SSL) that aims to alleviate this limitation by creating domain specific pretext tasks on unlabeled data. Simultaneously, there are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs). GNNs can naturally utilize unlabeled nodes through the simple neighborhood aggregation that is unable to thoroughly make use of unlabeled nodes. Thus, we seek to harness SSL for GNNs to fully exploit the unlabeled data. Different from data instances in the image and text domains, nodes in graphs present unique structure information and they are inherently linked indicating not independent and identically distributed (or i.i.d.). Such complexity is a double-edged sword for SSL on graphs. On the one hand, it determines that it is challenging to adopt solutions from the image and text domains to graphs and dedicated efforts are desired. On the other hand, it provides rich information that enables us to build SSL from a variety of perspectives. Thus, in this paper, we first deepen our understandings on when, why, and which strategies of SSL work with GNNs by empirically studying numerous basic SSL pretext tasks on graphs. Inspired by deep insights from the empirical studies, we propose a new direction SelfTask to build advanced pretext tasks that are able to achieve state-of-the-art performance on various real-world datasets. The specific experimental settings to reproduce our results can be found in \url{https://github.com/ChandlerBang/SelfTask-GNN}.

研究の動機と目的

• Understand when, why, and which SSL strategies work for GNNs on node classification.
• Systematically compare basic SSL pretext tasks based on graph structure and node attributes.
• Investigate training strategies (joint vs. two-stage) for integrating SSL with GNNs.
• Propose insights to inspire advanced pretext task designs (SelfTask).

提案手法

• Define graph SSL problem and present basic pretext tasks using structure (A) and attributes (X).
• Evaluate local/global structure-based SSL tasks: NodeProperty, EdgeMask, PairwiseDistance, Distance2Clusters.
• Evaluate attribute-based SSL tasks: AttributeMask, PairwiseAttrSim.
• Study joint training and two-stage training architectures for combining L.self with L.task.
• Empirically analyze on Cora, Citeseer, and Pubmed with GCN variants.
• Provide empirical guidelines on which SSL tasks and training strategy yield improvements.

実験結果

リサーチクエスチョン

• RQ1 Which SSL pretext tasks on graphs most improve node classification with GNNs?
• RQ2 How do local vs. global structure and attribute-based SSL tasks compare in effectiveness?
• RQ3 Which training paradigm (joint vs. two-stage) best integrates SSL with GNNs?
• RQ4 Can new task design principles (self-supervised insights) guide more effective SSL for graphs?

主な発見

モデル	結合訓練 - Cora	結合訓練 - Citeseer	結合訓練 - Pubmed	二段階訓練 - Cora	二段階訓練 - Citeseer	二段階訓練 - Pubmed
GCN	81.32	71.53	79.28	81.32	71.53	79.28
GCN-DroppedGraph	81.03	71.29	79.28	81.03	71.29	79.26
GCN-PCA	81.74	70.38	78.83	81.74	70.38	78.83
NodeProperty	81.94	71.60	79.44	81.59	71.69	79.24
EdgeMask	81.69	71.51	78.90	81.44	71.57	79.33
PairwiseNodeDistance	83.11	71.90	80.05	82.39	72.02	79.57
Distance2Cluster	83.55	71.44	79.88	81.80	71.55	79.51
AttributeMask	81.47	70.57	78.88	81.31	70.40	78.72
PairwiseAttrSim	83.05	71.67	79.45	81.57	71.74	79.42

• Global structure-based SSL tasks (e.g., PairwiseDistance, Distance2Clusters) consistently improve accuracy across datasets.
• Certain local tasks (NodeProperty, EdgeMask, AttributeMask) show limited gains compared to global/pairwise methods.
• Joint training generally outperforms two-stage training in most settings and is simpler to tune.
• Two-stage training can yield strong results but is sensitive and often requires more tuning.
• SSL-enabled GNNs outperform vanilla GCN on Cora, Citeseer, and Pubmed with several pretext tasks.
• Distance-to-cluster and pairwise attribute/similarity tasks offer notable gains (up to multiple percentage points in some cases).