[论文解读] Strategies for Pre-training Graph Neural Networks

Weihua Hu, Bowen Liu+5 more|arXiv (Cornell University)|May 29, 2019

Advanced Graph Neural Networks参考文献 62被引用 186

一句话总结

本文提出一种用于 GNNs 的双层级预训练策略——节点级自监督任务和图级监督任务——结合一个表达力强的 GNN (GIN) 以提升 OOD 泛化并达到最先进的结果，避免负迁移。

ABSTRACT

Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and then fine-tune it on a downstream task of interest. While pre-training has been effective in many language and vision domains, it remains an open question how to effectively use pre-training on graph datasets. In this paper, we develop a new strategy and self-supervised methods for pre-training Graph Neural Networks (GNNs). The key to the success of our strategy is to pre-train an expressive GNN at the level of individual nodes as well as entire graphs so that the GNN can learn useful local and global representations simultaneously. We systematically study pre-training on multiple graph classification datasets. We find that naive strategies, which pre-train GNNs at the level of either entire graphs or individual nodes, give limited improvement and can even lead to negative transfer on many downstream tasks. In contrast, our strategy avoids negative transfer and improves generalization significantly across downstream tasks, leading up to 9.4% absolute improvements in ROC-AUC over non-pre-trained models and achieving state-of-the-art performance for molecular property prediction and protein function prediction.

研究动机与目标

• 解决在任务特定标签稀缺且存在分布漂移的图数据学习挑战。
• 评估天真预训练策略是否有助于或有害于下游的图任务。
• 开发结合节点级与图级的预训练方法以提升可迁移性。
• 在分子性质预测与蛋白质功能预测上证明有效性。
• 提供数据集和基准以支持对图进行大规模预训练研究。

提出的方法

• 通过 Context Prediction 和 Attribute Masking 开发节点级预训练以捕捉局部领域特定知识。
• 通过监督多任务预训练开发图级预训练以注入全局的图级信号。
• 将主 GNN 与辅助 Context GNN 共同优化，并对 Context Prediction 使用负采样。
• 以一个表达力强的 Graph Isomorphism Network (GIN) 作为骨干网络，以最大化预训练带来的益处。
• 顺序：先进行节点级自监督，然后进行图级监督预训练，最后进行端到端微调。
• 展示仅使用图级预训练或节点级预训练可能导致负迁移；联合方法可减轻这一问题。

实验结果

研究问题

• RQ1 预训练 GNN 是否能提升下游图分类和功能预测任务的性能？
• RQ2 节点级与图级预训练是否为图的迁移学习提供互补收益？
• RQ3 在各领域中，天真地只做图级预训练或只做节点级预训练是否有利或有害？
• RQ4 哪些 GNN 架构最能从所提预训练策略中获益？

主要发现

• 结合节点级与图级预训练的策略在非预训练模型基础上实现了高达 9.4% 绝对 ROC-AUC 的提升。
• 在分子性质预测和蛋白质功能预测任务上，带预训练的 GIN 取得了最先进的结果。
• 仅进行图级预训练可能在大量任务上造成负迁移；与节点级预训练结合后可避免这一问题。
• 天真地进行大规模图级多任务预训练在多数任务上收益有限，甚至可能降低下游性能。
• 与未预训练模型相比，预训练模型在微调阶段收敛速度快数量级级别地提升。