[论文解读] Why Propagate Alone? Parallel Use of Labels and Features on Graphs

Yangkun Wang, Jiarui Jin+8 more|arXiv (Cornell University)|Oct 14, 2021

Advanced Graph Neural Networks参考文献 20被引用 5

一句话总结

本文对图神经网络中广泛使用的“标签技巧”进行了理论分析，证明其随机标签分割可被简化为包含数据拟合项和图结构自适应正则化项的确定性训练目标。该方法在多种GNN架构和基准测试中提升了性能，包括OGB上的SOTA结果，通过实现端到端训练并消除了标签泄漏。

ABSTRACT

Graph neural networks (GNNs) and label propagation represent two interrelated modeling strategies designed to exploit graph structure in tasks such as node property prediction. The former is typically based on stacked message-passing layers that share neighborhood information to transform node features into predictive embeddings. In contrast, the latter involves spreading label information to unlabeled nodes via a parameter-free diffusion process, but operates independently of the node features. Given then that the material difference is merely whether features or labels are smoothed across the graph, it is natural to consider combinations of the two for improving performance. In this regard, it has recently been proposed to use a randomly-selected portion of the training labels as GNN inputs, concatenated with the original node features for making predictions on the remaining labels. This so-called label trick accommodates the parallel use of features and labels, and is foundational to many of the top-ranking submissions on the Open Graph Benchmark (OGB) leaderboard. And yet despite its wide-spread adoption, thus far there has been little attempt to carefully unpack exactly what statistical properties the label trick introduces into the training pipeline, intended or otherwise. To this end, we prove that under certain simplifying assumptions, the stochastic label trick can be reduced to an interpretable, deterministic training objective composed of two factors. The first is a data-fitting term that naturally resolves potential label leakage issues, while the second serves as a regularization factor conditioned on graph structure that adapts to graph size and connectivity. Later, we leverage this perspective to motivate a broader range of label trick use cases, and provide experiments to verify the efficacy of these extensions.

研究动机与目标

• 对GNN中广泛采用但理解不足的标签技巧进行理论分析，该技巧通过随机分割训练标签以实现与特征的并行使用。
• 确定随机标签技巧是否可被简化为一种确定性、可解释的训练目标。
• 研究标签技巧在实际训练过程中引入的正则化效应及其泛化影响。
• 将标签技巧扩展至原本不可微的方法（如标签传播和Correct and Smooth, C&S），以实现端到端训练。
• 在多种GNN架构和基准数据集（包括OGB和MAG240M-LSC）上验证该方法。

提出的方法

• 提出一种确定性的自排除简化形式的随机标签技巧，即从输入中排除部分标签，仅将其用于监督。
• 证明随机标签技巧等价于由数据拟合项和依赖图结构与连通性的正则化项组成的确定性目标。
• 推导出正则化项会根据图的大小和密度自适应调整，作为标签预测的平滑性先验。
• 将确定性形式应用于扩展标签技巧至非端到端方法（如Correct and Smooth, C&S），实现可训练的后处理。
• 引入C&S的可学习版本，整合标签技巧，实现平滑权重的端到端优化。
• 在多个基准数据集（包括Cora-full、PubMed、ogbn-arxiv和ogbn-products）上，对线性模型、GNN和基于GBDT的预测器验证该方法。

实验结果

研究问题

• RQ1 随机标签技巧能否被简化为一种确定性、可解释的训练目标？
• RQ2 标签技巧在训练过程中引入了哪些统计特性与正则化性质？
• RQ3 当训练期间的预测与推理阶段不一致时，标签技巧如何影响模型泛化？
• RQ4 标签技巧能否被扩展以实现此前不可微图方法（如标签传播和C&S）的端到端训练？
• RQ5 该确定性形式是否在多种GNN架构和真实世界基准上均提升性能？

主要发现

• 随机标签技巧在理论上等价于一种确定性目标，包含防止标签泄漏的数据拟合项和依赖图结构的自适应正则化项。
• 在PubMed和ogbn-arxiv数据集上，确定性标签技巧在SGC、SIGN和TWIRLS模型上均提升了性能，分别取得最高1.8%和2.8%的准确率增益。
• 在Cora-full数据集上，性能与基线相当，因为模型已实现接近100%的训练准确率，表明当标签信息已充分嵌入时，收益趋于饱和。
• 对于线性模型，分割概率α可作为正则化系数，α越大性能越好；而对于GNN，由于其固有的非线性，最优α值较低。
• 可学习的C&S变体在Cora-full和PubMed上的验证准确率优于原始C&S，在ogbn-arxiv和ogbn-products上表现相当或更优。
• 当与GBDT结合时，标签技巧在表格型节点回归数据集上显著提升性能，证明其在标准GNN之外也具有广泛适用性。