[Paper Review] PairNorm: Tackling Oversmoothing in GNNs

Lingxiao Zhao, Leman Akoglu|arXiv (Cornell University)|Sep 26, 2019

Advanced Graph Neural Networks19 references158 citations

TL;DR

PairNorm introduces a fast, parameter-free normalization layer that prevents oversmoothing in deep GNNs, enabling deeper models (GCN, GAT, SGC) without performance loss and showing benefits in missing-feature settings.

ABSTRACT

The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PairNorm makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs. Code is available at https://github.com/LingxiaoShawn/PairNorm.

Motivation & Objective

• Investigate the oversmoothing phenomenon in deep GNNs and quantify its impact on node representations.
• Propose a universal normalization layer that prevents distant node embeddings from becoming too similar.
• Develop a simple, architecture-agnostic, parameter-free method that can be inserted between GNN layers.
• Demonstrate the effectiveness of PairNorm on real-world graphs and under missing-feature scenarios (SSNC-MV).

Proposed method

• Analyze graph convolution as a form of Laplacian smoothing and relate it to a graph-regularized least squares view (GRLS).
• Define PairNorm as a two-step center-and-scale normalization that preserves a constant total pairwise squared distance (TPSD) across layers.
• Center by subtracting the mean representation, then scale so that the TPSD remains constant (controlled by a hyperparameter s).
• Provide an optional PairNorm-si variant that enforces a fixed L2 norm per node.
• Apply PairNorm after each graph convolution layer (except the last) and evaluate on SGC, GCN, and GAT; compare with residual connections.
• Discuss a real-world scenario (SSNC-MV) where deeper GNNs become beneficial and PairNorm shows gains.

Experimental results

Research questions

• RQ1 How does oversmoothing cause representations to become indistinguishable as layers increase?
• RQ2 Can a simple, layer-wise normalization prevent cross-cluster feature mixing without changing network architecture?
• RQ3 Does PairNorm enable deeper versions of SGC, GCN, and GAT to maintain or improve accuracy?
• RQ4 How does PairNorm perform in semi-supervised node classification with missing features (SSNC-MV) across datasets?
• RQ5 In what settings do deeper GNNs with PairNorm provide the most benefit (e.g., missing-feature scenarios)?

Key findings

• PairNorm makes deep GCN, GAT, and SGC models more robust to oversmoothing, slowing performance degradation as layers grow.
• In missing-feature settings, PairNorm enables deeper models to recover performance and outperform vanilla baselines, especially at higher missing rates.
• PairNorm is fast, simple to implement, and introduces no additional parameters, functioning as a general normalization layer between layers.
• PairNorm-si generally yields stable improvements for GCN and GAT, while standard PairNorm works well for SGC.
• In common benchmark datasets, very deep networks offer limited absolute gains, but PairNorm enables deeper networks to be advantageous in realistic scenarios with missing features.
• The study also explores SSNC-MV, showing that deeper architectures with PairNorm can outperform shallower vanilla models under feature absence.