[Paper Review] GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

Tianle Cai, Shengjie Luo+4 more|arXiv (Cornell University)|Sep 7, 2020

Advanced Graph Neural NetworksComputer Science71 references74 citations

TL;DR

GraphNorm improves GNN training by adding a learnable shift to Graph InstanceNorm, mitigating information loss on regular graphs, yielding faster convergence and better generalization than BatchNorm, LayerNorm, InstanceNorm, or no normalization.

ABSTRACT

Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.

Motivation & Objective

• Motivate and understand normalization needs in Graph Neural Networks (GNNs).
• Evaluate existing normalization methods (BatchNorm, LayerNorm, InstanceNorm) when applied to GNNs.
• Identify limitations of InstanceNorm and BatchNorm in graphs.
• Propose GraphNorm to overcome expressiveness degradation while retaining preconditioning benefits.
• Demonstrate that GraphNorm accelerates training and improves generalization across benchmarks.

Proposed method

• Adapt BatchNorm, LayerNorm, and InstanceNorm to GNNs by applying normalization after linear/aggregation steps.
• Analyze why InstanceNorm shifts act as a preconditioner for GNN aggregation and why BatchNorm suffers from graph batch noise.
• Show that the standard InstanceNorm shift can degrade expressiveness on highly regular graphs.
• Introduce GraphNorm with a learnable shift parameter per feature dimension to preserve graph information.
• Provide theoretical intuition and empirical validation across eight graph classification benchmarks.
• Compare GraphNorm against BatchNorm, InstanceNorm, LayerNorm, and no normalization.

Experimental results

Research questions

• RQ1 How do standard normalization schemes (BatchNorm, LayerNorm, InstanceNorm) perform when integrated into GNN architectures?
• RQ2 Why does InstanceNorm accelerate GNN training, and why can its shift hurt expressiveness on regular graphs?
• RQ3 Can a learnable shift in graph normalization preserve critical graph information while retaining acceleration benefits?
• RQ4 Does GraphNorm improve convergence speed and generalization across diverse graph datasets and architectures (GIN/GCN)?

Key findings

• InstanceNorm speeds up GNN training more than BatchNorm or LayerNorm by acting as a preconditioner for graph aggregation.
• BatchNorm suffers from heavy batch noise on graph data, making it less effective for GNN optimization.
• A fixed shift in InstanceNorm can degrade expressiveness on highly regular graphs by removing informative mean statistics.
• GraphNorm introduces a learnable shift per feature dimension, preserving essential mean information and improving convergence.
• Empirically, GraphNorm outperforms other normalization methods in convergence speed and generalization on multiple graph benchmarks.