[论文解读] Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs

Chenxiao Yang, Qitian Wu+2 more|arXiv (Cornell University)|Dec 18, 2022

Advanced Graph Neural Networks被引用 16

一句话总结

本文提出了传播型多层感知机（PMLPs），这是一种新型模型类别，其训练方式与标准MLP相同，但在推理阶段应用消息传递，其架构与GNN相似。关键发现是，PMLPs在节点分类任务中始终与MLP或GNN表现相当甚至更优，表明GNN优越泛化能力的根源并非训练动态，而是推理阶段消息传递所引入的归纳偏置，从根本上重塑了对GNN泛化能力的理解。

ABSTRACT

Graph neural networks (GNNs), as the de-facto model class for representation learning on graphs, are built upon the multi-layer perceptrons (MLP) architecture with additional message passing layers to allow features to flow across nodes. While conventional wisdom commonly attributes the success of GNNs to their advanced expressivity, we conjecture that this is not the main cause of GNNs' superiority in node-level prediction tasks. This paper pinpoints the major source of GNNs' performance gain to their intrinsic generalization capability, by introducing an intermediate model class dubbed as P(ropagational)MLP, which is identical to standard MLP in training, but then adopts GNN's architecture in testing. Intriguingly, we observe that PMLPs consistently perform on par with (or even exceed) their GNN counterparts, while being much more efficient in training. This finding sheds new insights into understanding the learning behavior of GNNs, and can be used as an analytic tool for dissecting various GNN-related research problems. As an initial step to analyze the inherent generalizability of GNNs, we show the essential difference between MLP and PMLP at infinite-width limit lies in the NTK feature map in the post-training stage. Moreover, by examining their extrapolation behavior, we find that though many GNNs and their PMLP counterparts cannot extrapolate non-linear functions for extremely out-of-distribution samples, they have greater potential to generalize to testing samples near the training data range as natural advantages of GNN architectures.

研究动机与目标

• 探究GNN相较于MLP的泛化优势是否源于其架构或训练过程。
• 通过解耦训练与推理架构，分离消息传递在GNN性能中的作用。
• 挑战传统观点，即GNN的成功源于更强的表征能力，转而将其归因于推理阶段的归纳偏置。
• 提出一种新的分析框架——PMLPs，用于剖析GNN行为并理解图神经网络的泛化机制。

提出的方法

• 提出PMLP，一种在训练阶段使用标准MLP（相同架构、数据、损失函数、优化器），但在推理阶段应用非参数化消息传递层的模型。
• 在图结构数据上训练标准MLP，然后在测试阶段应用GNN风格的消息传递，形成PMLP。
• 在多个基准数据集和架构变体上，将PMLP性能与原始MLP和完整GNN进行对比。
• 分析PMLP和MLP在无限宽度极限下的神经正切核（NTK）特征图差异，以识别训练后两者之间的区别。
• 在分布偏移条件下，研究泛化与外推行为，特别是靠近训练数据支持区域的表现。
• 使用t-SNE可视化比较MLP、PMLP和GNN学习到的内部节点表征，评估类别可分性。

实验结果

研究问题

• RQ1 GNN中消息传递机制是否本身能提升泛化能力，而独立于训练动态？
• RQ2 一个以MLP方式训练但测试时使用消息传递的模型（PMLP）能否达到与完整GNN相当的性能？
• RQ3 GNN相较于MLP的泛化优势主要源于表征能力还是推理阶段的归纳偏置？
• RQ4 PMLP与GNN在处理分布外样本时，尤其是在靠近训练分布区域时，外推能力有何差异？
• RQ5 在无限宽度下，PMLP与MLP的NTK特征图有何区别？这种差异如何与泛化能力相关联？

主要发现

• 尽管共享相同的已训练权重，PMLPs在测试准确率上始终优于原始MLP，表明推理阶段的消息传递本身能显著提升泛化能力。
• 在噪声或稀疏化图设置下，PMLPs在节点分类任务中的表现与对应GNN相当甚至更优。
• PMLP的泛化差距显著小于MLP，证明GNN架构在推理阶段的归纳偏置能有效缓解过拟合。
• 在无限宽度极限下，PMLP的NTK特征图与MLP存在结构性差异，揭示了消息传递机制的内在影响。
• PMLP与GNN在靠近训练分布的样本上均表现出更平滑的过渡和更好的泛化能力，而MLP则在支持区域外出现突变线性化。
• t-SNE可视化结果证实，尽管PMLP在训练阶段使用与MLP相同的权重，其学习到的节点表征在判别性和类别可分性上均优于MLP，与GNN表现一致。