[论文解读] Self-supervised Learning on Graphs: Deep Insights and New Direction

Wei Jin, Tyler Derr+5 more|arXiv (Cornell University)|Jun 17, 2020

Advanced Graph Neural Networks参考文献 32被引用 111

一句话总结

本文分析了各种自监督学习任务如何影响用于节点分类的 GNNs，比较了联合训练与两阶段训练，并提出了可促成高级预文本任务（SelfTask）的见解。

ABSTRACT

The success of deep learning notoriously requires larger amounts of costly annotated data. This has led to the development of self-supervised learning (SSL) that aims to alleviate this limitation by creating domain specific pretext tasks on unlabeled data. Simultaneously, there are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs). GNNs can naturally utilize unlabeled nodes through the simple neighborhood aggregation that is unable to thoroughly make use of unlabeled nodes. Thus, we seek to harness SSL for GNNs to fully exploit the unlabeled data. Different from data instances in the image and text domains, nodes in graphs present unique structure information and they are inherently linked indicating not independent and identically distributed (or i.i.d.). Such complexity is a double-edged sword for SSL on graphs. On the one hand, it determines that it is challenging to adopt solutions from the image and text domains to graphs and dedicated efforts are desired. On the other hand, it provides rich information that enables us to build SSL from a variety of perspectives. Thus, in this paper, we first deepen our understandings on when, why, and which strategies of SSL work with GNNs by empirically studying numerous basic SSL pretext tasks on graphs. Inspired by deep insights from the empirical studies, we propose a new direction SelfTask to build advanced pretext tasks that are able to achieve state-of-the-art performance on various real-world datasets. The specific experimental settings to reproduce our results can be found in \url{https://github.com/ChandlerBang/SelfTask-GNN}.

研究动机与目标

• 理解何时、为何以及哪些自监督学习策略在节点分类的 GNNs 上有效。
• 系统性比较基于图结构和节点属性的基础 SSL 预文本任务。
• 研究将自监督学习与 GNNs 融合的训练策略（联合训练 vs. 两阶段）。
• 提出可启发高级预文本任务设计（SelfTask）的见解。

提出的方法

• 定义图的自监督学习问题，并呈现使用结构（A）和属性（X）的基本预文本任务。
• 评估基于局部/全局结构的自监督任务：NodeProperty、EdgeMask、PairwiseDistance、Distance2Clusters。
• 评估基于属性的自监督任务：AttributeMask、PairwiseAttrSim。
• 研究将自监督损失与任务损失相结合的联合训练与两阶段训练架构。
• 在 Cora、Citeseer 和 Pubmed 上使用不同的 GCN 变体进行经验分析。
• 提供关于哪些 SSL 任务和训练策略能带来改进的实证指南。

实验结果

研究问题

• RQ1 在图上哪些自监督预文本任务能最大程度提升 GNN 的节点分类性能？
• RQ2 局部与全局结构以及基于属性的自监督任务在效果上有何差异？
• RQ3 哪种训练范式（联合训练还是两阶段）能最好地将自监督学习与 GNNs 集成？
• RQ4 新的任务设计原则（自监督洞见）能否引导更有效的图自监督学习？

主要发现

• 基于全局结构的自监督任务（例如 PairwiseDistance、Distance2Clusters）在所有数据集上均能稳定提升准确率。
• 某些局部任务（NodeProperty、EdgeMask、AttributeMask）相比全局/成对方法收益有限。
• 联合训练在大多数设置中通常优于两阶段训练，且更易于调参。
• 两阶段训练也能取得强效果，但对超参较敏感，通常需要更多调参。
• 在 Cora、Citeseer 和 Pubmed 上，使用多种预文本任务的自监督 GNN 超越了原生 GCN。
• Distance-to-cluster 和成对属性/相似性任务提供显著提升（在某些情况下达到若干百分点）。