[Paper Review] A Note on Over-Smoothing for Graph Neural Networks

Chen Cai, Yusu Wang|arXiv (Cornell University)|Jun 23, 2020

Advanced Graph Neural Networks19 references96 citations

TL;DR

The paper analyzes over-smoothing in nonlinear GNNs using Dirichlet energy, showing embeddings can lose discriminative power as layers grow, and provides an alternative proof handling Leaky ReLU and varying dimensions, plus experiments on edge operations.

ABSTRACT

Graph Neural Networks (GNNs) have achieved a lot of success on graph-structured data. However, it is observed that the performance of graph neural networks does not improve as the number of layers increases. This effect, known as over-smoothing, has been analyzed mostly in linear cases. In this paper, we build upon previous results \cite{oono2019graph} to further analyze the over-smoothing effect in the general graph neural network architecture. We show when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power. Using Dirichlet energy to measure "expressiveness" of embedding is conceptually clean; it leads to simpler proofs than \cite{oono2019graph} and can handle more non-linearities.

Motivation & Objective

• Motivate the study of over-smoothing in deep GNNs and its impact on discriminative power.
• Extend prior linear analyses to nonlinear GNNs with general activations.
• Provide a Dirichlet-energy based framework to analyze deep GNNs and demonstrate exponential energy decay under certain conditions.
• Demonstrate the method’s applicability to Leaky ReLU and varying embedding dimensions.

Proposed method

• Represent each GCN layer as f_l(X) = ML P_l (P X) with augmented normalized Laplacian.
• Track Dirichlet energy E(X) = tr(X^T \tilde{Δ} X) across layers to analyze expressiveness.
• Prove E(PX) ≤ (1−λ)^2 E(X) where λ is the smallest non-zero eigenvalue of the augmented Laplacian.
• Show E(XW) ≤ ||W^T||_2^2 E(X) to bound the effect of linear mappings between layers.
• Prove E(σ(X)) ≤ E(X) for activations with Lipschitz constant ≤ 1, including ReLU and Leaky ReLU, enabling nonlinearity handling beyond ReLU.
• Derive a main result E(f_l(X)) ≤ s_l ¯λ E(X) and Corollary E(X^(l)) ≤ O((s ¯λ)^l) indicating exponential energy decay.

Experimental results

Research questions

• RQ1 Does over-smoothing cause exponential decay of Dirichlet energy in deep GNNs with nonlinear activations?
• RQ2 Can Dirichlet energy provide a clean, generalizable measure of embedding expressiveness for nonlinear GNNs beyond ReLU?
• RQ3 How do edge operations and varying embedding dimensions affect the Dirichlet energy and hence over-smoothing?
• RQ4 Is Leaky ReLU and other nonlinearities tractable within a Dirichlet-energy based analysis?

Key findings

• Dirichlet energy decreases exponentially with the number of layers under certain weight and spectral conditions, leading to loss of discriminative power.
• The Dirichlet-energy based approach extends to Leaky ReLU and handles varying embedding dimensions unlike some prior work.
• For regular graphs, the analysis applies to common nonlinearities beyond ReLU and provides an elementary linear-algebraic proof framework.
• Empirical results on various graphs show that dropping edges often increases Dirichlet energy, and increasing a few edge weights to a very high value can have a similar effect to edge dropping.