[论文解读] Effective Training Strategies for Deep Graph Neural Networks.

Kuangqi Zhou, Yanfei Dong+4 more|arXiv (Cornell University)|Jun 12, 2020

Advanced Graph Neural Networks参考文献 1被引用 21

一句话总结

本文识别出训练困难和过拟合——而不仅仅是过度平滑——是深层图卷积网络（GCNs）性能退化的主要原因。本文提出节点归一化（NodeNorm），通过使用每个节点自身的统计信息对节点进行归一化，以减少隐藏表征中的特征相关性，提升模型平滑性并稳定训练过程，从而使深层图神经网络（GNNs）的性能超越浅层模型。

ABSTRACT

Graph Neural Networks (GNNs) tend to suffer performance degradation as model depth increases, which is usually attributed in previous works to the oversmoothing problem. However, we find that although oversmoothing is a contributing factor, the main reasons for this phenomenon are training difficulty and overfitting, which we study by experimentally investigating Graph Convolutional Networks (GCNs), a representative GNN architecture. We find that training difficulty is caused by gradient vanishing and can be solved by adding residual connections. More importantly, overfitting is the major obstacle for deep GCNs and cannot be effectively solved by existing regularization techniques. Deep GCNs also suffer training instability, which slows down the training process. To address overfitting and training instability, we propose Node Normalization (NodeNorm), which normalizes each node using its own statistics in model training. The proposed NodeNorm regularizes deep GCNs by discouraging feature-wise correlation of hidden embeddings and increasing model smoothness with respect to input node features, and thus effectively reduces overfitting. Additionally, it stabilizes the training process and hence speeds up the training. Extensive experiments demonstrate that our NodeNorm method generalizes well to other GNN architectures, enabling deep GNNs to compete with and even outperform shallow ones. Code is publicly available.

研究动机与目标

• 探究深层图神经网络性能退化现象的根源，超越过度平滑的范畴。
• 识别出训练困难和过拟合是训练深层GCNs的主要障碍。
• 开发一种正则化技术，有效缓解深层GNN中的过拟合问题。
• 提升深层GCN架构的训练稳定性和速度。
• 提出一种可泛化至多种GNN架构的方法。

提出的方法

• 提出节点归一化（NodeNorm），在训练过程中利用每个节点自身的统计信息对隐藏表征进行归一化。
• 应用节点级批量归一化，对每个节点在批量中计算均值和方差，以降低隐藏表征中的特征相关性。
• 引入一种独立作用于每个节点的归一化层，增强模型对输入特征的平滑性。
• 使用残差连接以缓解梯度消失问题，提升训练稳定性。
• 将残差连接与NodeNorm结合，同时应对梯度消失与过拟合问题。
• 采用一种训练目标，通过节点特定的统计信息促进隐藏表征的平滑性与去相关性。

实验结果

研究问题

• RQ1 除了过度平滑之外，深层GCNs性能退化的主要原因是什么？
• RQ2 训练困难和过拟合在多大程度上导致了深层GNN的性能下降？
• RQ3 现有正则化技术能否有效缓解深层GCNs中的过拟合？
• RQ4 节点级归一化在多大程度上提升了深层GNN的训练稳定性和泛化能力？
• RQ5 所提出的NodeNorm方法是否在不同GNN架构中具有良好的泛化能力？

主要发现

• 深层GCNs中的训练困难主要源于梯度消失，而残差连接可有效缓解该问题。
• 过拟合是深层GCNs中的主要障碍，且标准正则化技术难以充分应对。
• NodeNorm通过抑制隐藏表征中的特征相关性，显著减少过拟合。
• NodeNorm增强了模型对输入节点特征的平滑性，从而改善泛化性能。
• 残差连接与NodeNorm的结合有效稳定了训练过程并加速了收敛。
• 大量实验表明，采用NodeNorm的深层GNN在多个基准测试和架构中均优于浅层GNN。