[Paper Review] Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Again

Ajay Jaiswal, Peihao Wang+4 more|arXiv (Cornell University)|Oct 14, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes that poor gradient flow, not just over-smoothing, undermines deep vanilla-GCNs. By introducing a topology-aware isometric initialization and gradient-guided dynamic rewiring with skip connections, the authors significantly improve gradient flow, enabling vanilla-GCNs to achieve state-of-the-art performance on citation networks without architectural complexity.

ABSTRACT

Despite the enormous success of Graph Convolutional Networks (GCNs) in modeling graph-structured data, most of the current GCNs are shallow due to the notoriously challenging problems of over-smoothening and information squashing along with conventional difficulty caused by vanishing gradients and over-fitting. Previous works have been primarily focused on the study of over-smoothening and over-squashing phenomena in training deep GCNs. Surprisingly, in comparison with CNNs/RNNs, very limited attention has been given to understanding how healthy gradient flow can benefit the trainability of deep GCNs. In this paper, firstly, we provide a new perspective of gradient flow to understand the substandard performance of deep GCNs and hypothesize that by facilitating healthy gradient flow, we can significantly improve their trainability, as well as achieve state-of-the-art (SOTA) level performance from vanilla-GCNs. Next, we argue that blindly adopting the Glorot initialization for GCNs is not optimal, and derive a topology-aware isometric initialization scheme for vanilla-GCNs based on the principles of isometry. Additionally, contrary to ad-hoc addition of skip-connections, we propose to use gradient-guided dynamic rewiring of vanilla-GCNs} with skip connections. Our dynamic rewiring method uses the gradient flow within each layer during training to introduce on-demand skip-connections adaptively. We provide extensive empirical evidence across multiple datasets that our methods improve gradient flow in deep vanilla-GCNs and significantly boost their performance to comfortably compete and outperform many fancy state-of-the-art methods. Codes are available at: https://github.com/VITA-Group/GradientGCN.

Motivation & Objective

• To investigate why deep vanilla-GCNs underperform despite theoretical expressivity, focusing on poor gradient flow during backpropagation.
• To challenge the common practice of using Glorot initialization in GCNs, which can harm gradient flow in deep architectures.
• To develop a principled method for dynamically injecting skip-connections based on layer-wise gradient flow to improve trainability.
• To demonstrate that simple vanilla-GCNs can achieve SOTA performance when equipped with improved gradient flow mechanisms.

Proposed method

• Derive a topology-aware isometric initialization for vanilla-GCNs using graph Laplacian and degree information to maintain signal stability across layers.
• Define gradient flow using the p-norm of gradients within each layer as a metric to identify poorly trained layers during training.
• Propose a gradient-guided dynamic rewiring mechanism that adaptively adds skip-connections only to layers with weak gradient signals.
• Integrate the new initialization and dynamic rewiring into standard GCN training without modifying the core architecture or adding extra parameters.
• Use Dirichlet energy as a proxy to measure embedding expressiveness and track how the methods preserve it during training.
• Evaluate the methods across multiple datasets (Cora, PubMed, Citeseer) using standard GCN setups and SOTA baselines.

Experimental results

Research questions

• RQ1 Does poor gradient flow significantly impair the trainability and performance of deep vanilla-GCNs?
• RQ2 Can a topology-aware initialization improve gradient flow and performance in vanilla-GCNs compared to standard Glorot initialization?
• RQ3 Is dynamic skip-connection injection based on gradient flow more effective than static or ad-hoc skip-connections in deep GCNs?
• RQ4 Can vanilla-GCNs achieve SOTA performance when equipped with improved gradient flow mechanisms?

Key findings

• The proposed topology-aware isometric initialization significantly improves gradient flow in deep vanilla-GCNs, leading to faster convergence and better performance on all benchmark datasets.
• Gradient-guided dynamic rewiring successfully identifies and strengthens poorly trained layers, reducing the number of layers with near-zero gradients during backpropagation.
• The combination of improved initialization and dynamic rewiring maintains Dirichlet energy across layers, preventing the loss of embedding expressiveness during training.
• On Cora, PubMed, and Citeseer, the method achieves state-of-the-art or near-SOTA performance using only vanilla-GCNs, outperforming many complex GNN architectures.
• The method reduces the exponential decay of Dirichlet energy observed in standard GCNs, preserving node representation diversity and expressiveness in deep layers.