[Paper Review] When Does Self-Supervision Help Graph Convolutional Networks?

Yuning You, Tianlong Chen+2 more|arXiv (Cornell University)|Jun 16, 2020

Advanced Graph Neural Networks53 references65 citations

TL;DR

The paper systematically studies incorporating self-supervision into GCNs via three schemes and three tasks, showing multi-task self-supervision yields the strongest generalization and robustness gains across datasets and architectures.

ABSTRACT

Self-supervision as an emerging technique has been employed to train convolutional neural networks (CNNs) for more transferrable, generalizable, and robust representation learning of images. Its introduction to graph convolutional networks (GCNs) operating on graph data is however rarely explored. In this study, we report the first systematic exploration and assessment of incorporating self-supervision into GCNs. We first elaborate three mechanisms to incorporate self-supervision into GCNs, analyze the limitations of pretraining & finetuning and self-training, and proceed to focus on multi-task learning. Moreover, we propose to investigate three novel self-supervised learning tasks for GCNs with theoretical rationales and numerical comparisons. Lastly, we further integrate multi-task self-supervision into graph adversarial training. Our results show that, with properly designed task forms and incorporation mechanisms, self-supervision benefits GCNs in gaining more generalizability and robustness. Our codes are available at https://github.com/Shen-Lab/SS-GCNs.

Motivation & Objective

• Assess whether self-supervised learning improves GCN classification performance.
• Identify which self-supervised task designs are most beneficial for GCNs.
• Compare pretraining, self-training, and multi-task learning schemes for integrating self-supervision into GCNs.
• Evaluate whether self-supervision enhances adversarial robustness in graphs.

Proposed method

• Analyze three schemes to inject self-supervision into GCNs: pretraining & finetuning, self-training (M3S), and multi-task learning.
• Propose three GCN-specific self-supervised tasks: node clustering, graph partitioning, and graph completion.
• Formalize multi-task learning as a joint objective with a supervised loss and a self-supervised loss (alpha-weighted).
• Introduce a graph adversarial defense setup with self-supervision as a regularizer during adversarial training.
• Experiment across standard datasets (Cora, Citeseer, PubMed) and SOTA GNNs (GCN, GAT, GIN, GMNN, GraphMix) to evaluate generalization and robustness.

Experimental results

Research questions

• RQ1 Can self-supervised learning improve GCN classification performance?
• RQ2 Which self-supervised pretext tasks are useful for GCNs?
• RQ3 Does the design of self-supervised tasks affect GCN performance across datasets and architectures?
• RQ4 Does self-supervision improve adversarial robustness of GCNs?
• RQ5 How do different self-supervision schemes compare in practice?

Key findings

• Multi-task learning consistently outperforms pretraining & finetuning and self-training for GCNs across datasets.
• Graph partitioning (topology-based) and graph completion (context-based) provide strong gains, with task effectiveness varying by dataset and architecture.
• Different GNN architectures (GCN, GAT, GIN) benefit from multi-task self-supervision, while some SLAs (GMNN, GraphMix) show mixed results depending on the task.
• Self-supervision via multi-task learning improves robustness to adversarial graph attacks without needing larger models or more data.
• Graph completion as a self-supervised task can offer substantial gains when neighborhood contexts are informative.