[Paper Review] Revisiting Over-smoothing in Deep GCNs

Chaoqi Yang, Ruijie Wang+3 more|arXiv (Cornell University)|Mar 30, 2020

Advanced Graph Neural Networks41 references56 citations

TL;DR

The paper reinterprets deep GCNs as alternating between graph-regularized MLP propagation and supervised training, showing deep GCNs can anti-oversmooth during training and introducing a cheap mean-subtraction trick to accelerate training.

ABSTRACT

Oversmoothing has been assumed to be the major cause of performance drop in deep graph convolutional networks (GCNs). In this paper, we propose a new view that deep GCNs can actually learn to anti-oversmooth during training. This work interprets a standard GCN architecture as layerwise integration of a Multi-layer Perceptron (MLP) and graph regularization. We analyze and conclude that before training, the final representation of a deep GCN does over-smooth, however, it learns anti-oversmoothing during training. Based on the conclusion, the paper further designs a cheap but effective trick to improve GCN training. We verify our conclusions and evaluate the trick on three citation networks and further provide insights on neighborhood aggregation in GCNs.

Motivation & Objective

• Clarify the role of oversmoothing in deep GCNs and distinguish training dynamics before and after learning.
• Propose a theoretical view of GCN as a two-step optimization combining graph regularization and supervised training.
• Identify practical techniques to accelerate and stabilize training of deep GCNs.
• Provide empirical insights into neighborhood aggregation and the effect of initialization on deep GCNs.

Proposed method

• Reformulate GCN as a two-step process: STEP1 minimizes a graph regularizer via layerwise propagation, embedding Lreg into the forward path; STEP2 trains the network by minimizing the empirical loss L0.
• Show that before training, deep GCNs oversmooth due to the forward propagation minimizing Lreg; during training, optimization of Wl mitigates oversmoothing (anti-oversmoothing) to fit L0.
• Derive that mean-subtraction per layer approximates the Fiedler vector, providing a coarse graph partition to speed up training.
• Analyze the role of the aggregation weight η and its relation to neighbor information weighting in η-GCN experiments.
• Compare mean-subtraction with PairNorm and BatchNorm, highlighting speed and stability benefits without extra parameters.

Experimental results

Research questions

• RQ1 Does oversmoothing fundamentally drive performance drops in deep GCNs, or can training dynamics counteract oversmoothing?
• RQ2 Can a simple trick (mean-subtraction) accelerate and stabilize training of deep GCNs without increasing model complexity?
• RQ3 How does the weight of neighborhood aggregation affect performance across shallow and deep GCNs?
• RQ4 What is the relationship between the two-step optimization view (graph regularization + supervised training) and observed training dynamics?

Key findings

• Deep GCNs exhibit initial oversmoothing before training but learn anti-oversmoothing during training, with overfitting proposed as a major factor for performance drops in very deep models.
• A mean-subtraction per layer substantially speeds up training and yields robust improvements, aligning with faster convergence and competitive test accuracy.
• The mean-subtraction trick approximates the Fiedler vector and provides a coarse graph partition, aiding training efficiency across datasets.
• In 2-layer GCNs, increasing the neighborhood aggregation weight improves test accuracy up to a point, after which gains saturate; in 32-layer GCNs, larger weights can maintain high training accuracy though test accuracy varies more, highlighting depth-dependent dynamics.
• SGC (linear GCN) exhibits oversmoothing with depth, while trained GCNs can avoid this via optimization of weights and nonlinearity, supporting the anti-oversmoothing view.
• Mean-subtraction outperforms PairNorm in test accuracy with faster training times and less variance.