[Paper Review] On Provable Benefits of Depth in Training Graph Convolutional Networks

Weilin Cong, Morteza Ramezani+1 more|arXiv (Cornell University)|Oct 28, 2021

Advanced Graph Neural NetworksComputer Science56 references17 citations

TL;DR

This paper challenges the conventional belief that depth harms Graph Convolutional Networks (GCNs) due to over-smoothing, proving that deeper GCNs are provably expressive and can achieve high training accuracy with proper training. It proposes a decoupled GCN architecture separating feature propagation from weight learning to preserve expressiveness and improve generalization, validated empirically on synthetic and real-world datasets.

ABSTRACT

Graph Convolutional Networks (GCNs) are known to suffer from performance degradation as the number of layers increases, which is usually attributed to over-smoothing. Despite the apparent consensus, we observe that there exists a discrepancy between the theoretical understanding of over-smoothing and the practical capabilities of GCNs. Specifically, we argue that over-smoothing does not necessarily happen in practice, a deeper model is provably expressive, can converge to global optimum with linear convergence rate, and achieve very high training accuracy as long as properly trained. Despite being capable of achieving high training accuracy, empirical results show that the deeper models generalize poorly on the testing stage and existing theoretical understanding of such behavior remains elusive. To achieve better understanding, we carefully analyze the generalization capability of GCNs, and show that the training strategies to achieve high training accuracy significantly deteriorate the generalization capability of GCNs. Motivated by these findings, we propose a decoupled structure for GCNs that detaches weight matrices from feature propagation to preserve the expressive power and ensure good generalization performance. We conduct empirical evaluations on various synthetic and real-world datasets to validate the correctness of our theory.

Motivation & Objective

• To investigate whether increasing depth in GCNs truly impairs their expressiveness due to over-smoothing, as commonly believed.
• To understand why deeper GCNs generalize poorly despite achieving high training accuracy when properly trained.
• To develop a theoretically grounded, decoupled GCN architecture that preserves expressive power and enhances generalization performance.
• To provide a novel generalization analysis for GCNs under semi-supervised node classification, identifying training strategy-induced degradation.
• To validate the theoretical claims through extensive empirical evaluation on synthetic and real-world graph datasets.

Proposed method

• Uses the Weisfeiler-Lehman (WL) graph isomorphism test to formally characterize the representational capacity of GCNs, showing deeper models are at least as expressive as shallow ones.
• Proves that deeper GCNs can converge to the global optimum with a linear convergence rate under proper training, implying over-smoothing is not the primary cause of performance degradation.
• Proposes a decoupled GCN structure that separates learnable weight matrices from the graph propagation mechanism to decouple expressiveness from optimization dynamics.
• Employs a generalization analysis framework under semi-supervised node classification to link poor generalization in deep GCNs to specific training strategies that degrade generalization capability.
• Derives theoretical bounds on gradient differences between perturbed and original models, showing that the generalization error is controlled by model depth and parameter sensitivity.
• Validates the theory via empirical evaluations on Cora, PubMed, Reddit, and synthetic datasets, comparing standard GCNs with the proposed decoupled variant.

Experimental results

Research questions

• RQ1 Does increasing the depth of GCNs truly impair their expressive power due to over-smoothing, or is this a misconception based on theoretical simplifications?
• RQ2 Why do deeper GCNs achieve high training accuracy but poor generalization performance, despite being capable of learning complex representations?
• RQ3 Can a structural modification to GCNs decouple expressiveness from optimization dynamics to improve generalization without sacrificing training accuracy?
• RQ4 How do training strategies such as dropout and weight decay affect the generalization capability of deep GCNs, and can this be formally analyzed?
• RQ5 To what extent does the proposed decoupled GCN architecture outperform standard GCNs in terms of generalization while maintaining high training accuracy?

Key findings

• Deeper GCNs are provably more expressive than shallow ones under the Weisfeiler-Lehman framework, as they can distinguish nodes with different neighborhood structures that shallow GCNs cannot.
• High training accuracy in deeper GCNs is achievable with sufficient training iterations, indicating that over-smoothing is not the root cause of performance degradation.
• The generalization performance of deeper GCNs deteriorates significantly due to training strategies like dropout and weight decay, which introduce regularization that harms generalization.
• The proposed decoupled GCN architecture, which separates feature propagation from weight learning, achieves better generalization performance while maintaining high training accuracy on multiple benchmark datasets.
• Empirical results on Cora, PubMed, Reddit, and synthetic graphs confirm that the decoupled architecture generalizes significantly better than standard GCNs, especially in deep settings.
• Theoretical analysis shows that gradient differences in deeper GCNs are bounded and depend on depth and parameter sensitivity, supporting the claim that generalization issues stem from training dynamics, not expressiveness limits.