[Paper Review] Tackling Over-Smoothing for General Graph Convolutional Networks

Wenbing Huang, Yu Rong+3 more|arXiv (Cornell University)|Aug 22, 2020

Advanced Graph Neural NetworksComputer Science46 references62 citations

TL;DR

The paper analyzes over-smoothing in general GCNs, proves convergence to a cuboid in deep networks, and proposes DropEdge to alleviate it with theory and experiments.

ABSTRACT

Increasing the depth of GCN, which is expected to permit more expressivity, is shown to incur performance detriment especially on node classification. The main cause of this lies in over-smoothing. The over-smoothing issue drives the output of GCN towards a space that contains limited distinguished information among nodes, leading to poor expressivity. Several works on refining the architecture of deep GCN have been proposed, but it is still unknown in theory whether or not these refinements are able to relieve over-smoothing. In this paper, we first theoretically analyze how general GCNs act with the increase in depth, including generic GCN, GCN with bias, ResGCN, and APPNP. We find that all these models are characterized by a universal process: all nodes converging to a cuboid. Upon this theorem, we propose DropEdge to alleviate over-smoothing by randomly removing a certain number of edges at each training epoch. Theoretically, DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by dimension collapse. Experimental evaluations on simulated dataset have visualized the difference in over-smoothing between different GCNs. Moreover, extensive experiments on several real benchmarks support that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs.

Motivation & Objective

• Explain why deeper GCNs suffer from over-smoothing across generic GCNs, GCN-b, ResGCN, and APPNP.
• Characterize the asymptotic behavior of deep GCNs under non-linearity.
• Propose DropEdge to mitigate over-smoothing and analyze its theoretical impact.
• Demonstrate empirical improvements of DropEdge on multiple node classification benchmarks.

Proposed method

• Define the augmented normalized adjacency and the subspace M spanned by its top eigenvectors.
• Prove a general over-smoothing theorem showing convergence to a cuboid O(M, r) for several GCN variants.
• Introduce DropEdge: randomly dropping edges with probability p during training and re-normalizing.
• Provide theoretical bounds showing how DropEdge increases the spectral radius bounds and slows convergence to over-smoothing.
• Show that DropEdge acts as a data augmentation and a message-passing reducer.
• Evaluate DropEdge on shallow and deep GCN backbones across multiple benchmarks.

Experimental results

Research questions

• RQ1 Why do general deep GCNs converge to a low-variance representation (over-smoothing) as depth increases?
• RQ2 How do variants like GCN-b, ResGCN, and APPNP differ in their convergence to subspaces or cuboids?
• RQ3 Can a simple edge-dropping strategy (DropEdge) theoretically and empirically alleviate over-smoothing across these models?
• RQ4 What is the impact of DropEdge on model expressivity and stability during training?

Key findings

• All four models converge to a cuboid rather than a subspace under infinite depth, with radius r depending on the model.
• GCN without bias converges to a zero-radius subspace, confirming over-smoothing.
• GCN with bias and APPNP converge to a cuboid with non-zero radius, slowing information loss.
• DropEdge increases the effective spectral bounds (lambda) and reduces the speed of over-smoothing.
• DropEdge serves as both a regularizer against overfitting and a mechanism to preserve information by enlarging the effective representational space.
• Empirical results show DropEdge improves performance on both shallow and deep GCN variants across several node classification benchmarks.