[Paper Review] Effective Training Strategies for Deep Graph Neural Networks.

Kuangqi Zhou, Yanfei Dong+4 more|arXiv (Cornell University)|Jun 12, 2020

Advanced Graph Neural NetworksComputer Science1 references21 citations

TL;DR

This paper identifies training difficulty and overfitting—not just oversmoothing—as the primary causes of performance degradation in deep Graph Convolutional Networks (GCNs). It proposes Node Normalization (NodeNorm), which normalizes each node using its own statistics to reduce feature-wise correlation in hidden embeddings, improve model smoothness, and stabilize training, enabling deep GNNs to outperform shallow ones.

ABSTRACT

Graph Neural Networks (GNNs) tend to suffer performance degradation as model depth increases, which is usually attributed in previous works to the oversmoothing problem. However, we find that although oversmoothing is a contributing factor, the main reasons for this phenomenon are training difficulty and overfitting, which we study by experimentally investigating Graph Convolutional Networks (GCNs), a representative GNN architecture. We find that training difficulty is caused by gradient vanishing and can be solved by adding residual connections. More importantly, overfitting is the major obstacle for deep GCNs and cannot be effectively solved by existing regularization techniques. Deep GCNs also suffer training instability, which slows down the training process. To address overfitting and training instability, we propose Node Normalization (NodeNorm), which normalizes each node using its own statistics in model training. The proposed NodeNorm regularizes deep GCNs by discouraging feature-wise correlation of hidden embeddings and increasing model smoothness with respect to input node features, and thus effectively reduces overfitting. Additionally, it stabilizes the training process and hence speeds up the training. Extensive experiments demonstrate that our NodeNorm method generalizes well to other GNN architectures, enabling deep GNNs to compete with and even outperform shallow ones. Code is publicly available.

Motivation & Objective

• To investigate the root causes of performance degradation in deep Graph Neural Networks beyond oversmoothing.
• To identify training difficulty and overfitting as the main obstacles in training deep GCNs.
• To develop a regularization technique that effectively mitigates overfitting in deep GNNs.
• To improve training stability and speed in deep GCN architectures.
• To propose a method that generalizes across various GNN architectures.

Proposed method

• Proposes Node Normalization (NodeNorm), which normalizes each node's hidden representation using its own statistics during training.
• Applies node-wise batch normalization that computes mean and variance per node across the batch, reducing feature-wise correlation in hidden embeddings.
• Introduces a normalization layer that operates independently per node, enhancing model smoothness with respect to input features.
• Uses residual connections to address gradient vanishing, improving training stability.
• Combines residual connections and NodeNorm to simultaneously tackle gradient vanishing and overfitting.
• Employs a training objective that encourages smoothness and decorrelation in hidden representations via node-specific statistics.

Experimental results

Research questions

• RQ1 What are the primary causes of performance degradation in deep GCNs beyond oversmoothing?
• RQ2 To what extent do training difficulty and overfitting contribute to the performance drop in deep GNNs?
• RQ3 Can overfitting in deep GCNs be effectively mitigated by existing regularization techniques?
• RQ4 How does node-wise normalization improve training stability and generalization in deep GNNs?
• RQ5 Does the proposed NodeNorm method generalize across different GNN architectures?

Key findings

• Training difficulty in deep GCNs is primarily caused by gradient vanishing, which is effectively mitigated by residual connections.
• Overfitting is the dominant obstacle in deep GCNs and cannot be adequately addressed by standard regularization techniques.
• NodeNorm significantly reduces overfitting by discouraging feature-wise correlation in hidden embeddings.
• NodeNorm enhances model smoothness with respect to input node features, improving generalization.
• The combination of residual connections and NodeNorm stabilizes training and accelerates convergence.
• Extensive experiments show that deep GNNs with NodeNorm can outperform shallow GNNs across multiple benchmarks and architectures.