[Paper Review] Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

Kenta Oono, Taiji Suzuki|arXiv (Cornell University)|May 27, 2019

Advanced Graph Neural Networks57 references184 citations

TL;DR

The paper shows that, as layers grow, Graph Convolutional Networks (GCNs) exponentially lose expressive power due to graph spectral properties, leading to information loss on dense graphs, and proposes weight normalization guidelines tested on real data.

ABSTRACT

Graph Neural Networks (graph NNs) are a promising deep learning approach for analyzing graph-structured data. However, it is known that they do not improve (or sometimes worsen) their predictive performance as we pile up many layers and add non-lineality. To tackle this problem, we investigate the expressive power of graph NNs via their asymptotic behaviors as the layer size tends to infinity. Our strategy is to generalize the forward propagation of a Graph Convolutional Network (GCN), which is a popular graph NN variant, as a specific dynamical system. In the case of a GCN, we show that when its weights satisfy the conditions determined by the spectra of the (augmented) normalized Laplacian, its output exponentially approaches the set of signals that carry information of the connected components and node degrees only for distinguishing nodes. Our theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra. To demonstrate this, we characterize the asymptotic behavior of GCNs on the Erdős -- Rényi graph. We show that when the Erdős -- Rényi graph is sufficiently dense and large, a broad range of GCNs on it suffers from the "information loss" in the limit of infinite layers with high probability. Based on the theory, we provide a principled guideline for weight normalization of graph NNs. We experimentally confirm that the proposed weight scaling enhances the predictive performance of GCNs in real data. Code is available at https://github.com/delta2323/gnn-asymptotics.

Motivation & Objective

• Understand how the expressive power of graph neural networks degrades as depth increases.
• Relate the asymptotic behavior of GCNs to the spectral properties of underlying graphs.
• Characterize information loss on dense graphs and Erdős–Rényi graphs.
• Provide principled guidelines for weight normalization to mitigate over-smoothing in graph NNs.

Proposed method

• Model forward propagation of a GCN as a dynamical system with an MLP composed over a linear operator P on node features.
• Define the invariant subspace M = U ⊗ R^C where U has non-negative orthonormal basis vectors and is P-invariant.
• Prove that d_M(f_l(X)) ≤ s_l λ d_M(X) where s_l is the product of layer weight singular values and λ relates to the spectrum of the augmented normalized Laplacian.
• Specialize the general result to GCNs with augmented normalized Laplacian P and show X^(l) converges toward M with rate (sλ)^l.
• Apply the theory to Erdős–Rényi graphs G_{N,p} to derive conditions under which information loss occurs with high probability.
• Provide a weight normalization guideline based on the spectral parameter λ and empirical verification on real data.

Experimental results

Research questions

• RQ1 Under what spectral conditions does a GCN lose its ability to distinguish nodes as depth increases?
• RQ2 How does the spectrum of the augmented normalized Laplacian govern information preservation or loss in GCNs?
• RQ3 Can a principled weight normalization counteract over-smoothing and improve deep GCN performance on real graphs?
• RQ4 Do Erdős–Rényi graphs exhibit depth-induced information loss, and how does graph density affect this behavior?

Key findings

• The output of a GCN tends toward an invariant subspace M tied to the lowest-frequency graph Laplacian components as depth grows.
• Distance to the invariant space obeys d_M(f_l(X)) ≤ (∏_h s_lh) λ d_M(X), implying exponential convergence to M when sλ < 1.
• On sufficiently dense and large Erdős–Rényi graphs, many GCNs suffer information loss with high probability as layers grow.
• There exists a weight-scaling threshold (sλ) that governs whether deep GCNs retain discriminative information or collapse toward the invariant space.
• Empirical experiments show weight normalization guided by the theory can improve predictive performance on real datasets (e.g., Cora synthetically dense graphs, and noisy versions of real graphs).
• The analysis links graph NN expressive power to topological graph information via the spectrum of the augmented normalized Laplacian.