[Paper Review] Towards Deeper Graph Neural Networks with Differentiable Group Normalization

Kaixiong Zhou, Xiao Huang+4 more|arXiv (Cornell University)|Jun 12, 2020

Advanced Graph Neural Networks25 references81 citations

TL;DR

This paper introduces differentiable group normalization (DGN) to alleviate over-smoothing in Graph Neural Networks, enabling deeper architectures and improved node classification by clustering nodes into groups and normalizing them independently. It also proposes two over-smoothing metrics—group distance ratio and instance information gain.

ABSTRACT

Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-optimal performance. The representations of nodes within the same community/class need be similar to facilitate the classification, while different classes are expected to be separated in embedding space. To bridge the gap, we introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN). It normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue. Experiments on real-world datasets demonstrate that DGN makes GNN models more robust to over-smoothing and achieves better performance with deeper GNNs.

Motivation & Objective

• Quantify over-smoothing in GNNs from group and instance perspectives using new metrics.
• Propose a differentiable group normalization technique to reduce over-smoothing.
• Demonstrate that DGN enables deeper GNNs with improved performance on benchmark datasets.
• Show robustness of DGN in scenarios with missing node features.

Proposed method

• Define two metrics to measure over-smoothing: Group Distance Ratio and Instance Information Gain.
• Introduce differentiable group normalization (DGN) that softly clusters nodes into groups and normalizes each group independently between GNN layers.
• Compute group assignments via a differentiable softmax-based clustering: S^(k) = softmax(H^(k) U^(k)).
• Normalize each group with its own running mean/variance and affine parameters, then combine with the original embeddings as H^(k) + λ sum_i tilde{H}^(k)_i.
• Train end-to-end to optimize both the supervised loss and the regularization effects implicit in the group normalization.
• Demonstrate that DGN preserves input features while disentangling group distributions to mitigate over-smoothing.

Experimental results

Research questions

• RQ1 How can over-smoothing in GNNs be precisely measured beyond pairwise node distances?
• RQ2 Can a group-wise normalization strategy mitigate over-smoothing without sacrificing useful self-features?
• RQ3 Does enabling deeper GNNs with DGN improve performance on standard benchmarks and in feature-mMissing scenarios?

Key findings

• DGN significantly alleviates over-smoothing and outperforms none, batch, and pair normalization across multiple datasets and depths.
• With DGN, deeper GNNs achieve higher accuracy than shallower counterparts, e.g., a highest accuracy of 79.7% at K = 20 for SGC on Cora.
• In the missing-feature scenario, DGN yields substantial gains over baselines: average improvements of 37.8% (over NN), 7.1% (over BN), and 12.8% (over PN).
• DGN enables deeper architectures to exploit multi-hop neighborhood information effectively, often using larger optimal layer counts (e.g., up to 30 layers in some setups).
• The method maintains a self-preserving component H^(k) to avoid over-normalization, while group-wise normalization disentangles distributions across groups.