[Paper Review] Simple and Deep Graph Convolutional Networks

Ming Chen, Zhewei Wei+3 more|arXiv (Cornell University)|Jul 4, 2020

Advanced Graph Neural NetworksComputer Science399 citations

TL;DR

GCNII extends vanilla GCN with initial residual and identity mapping to enable truly deep graph networks and alleviate over-smoothing, achieving state-of-the-art results on semi- and full-supervised tasks.

ABSTRACT

Graph convolutional networks (GCNs) are a powerful deep learning approach for graph-structured data. Recently, GCNs and subsequent variants have shown superior performance in various application areas on real-world datasets. Despite their success, most of the current GCN models are shallow, due to the {\em over-smoothing} problem. In this paper, we study the problem of designing and analyzing deep graph convolutional networks. We propose the GCNII, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping}. We provide theoretical and empirical evidence that the two techniques effectively relieves the problem of over-smoothing. Our experiments show that the deep GCNII model outperforms the state-of-the-art methods on various semi- and full-supervised tasks. Code is available at https://github.com/chennnM/GCNII .

Motivation & Objective

• Motivate the design of deep GCNs despite over-smoothing in shallow models.
• Propose two simple techniques—initial residual and identity mapping—to enable deep GCNs.
• Provide theoretical analysis of multi-layer GCNs and GCNII’s expressive power and convergence properties.
• Demonstrate empirical gains of GCNII over state-of-the-art methods across multiple datasets and depths.

Proposed method

• Introduce GCNII by adding initial residual connections from the input layer to each layer.
• Add an identity mapping term to the layer weight matrix to improve trainability and expressiveness.
• Train with a fixed renormalized propagation matrix tilde{P} and learn layer-wise coefficients through  parameters; incorporate two hyperparameters
   and  per layer.
• Theoretically analyze the stationary state and convergence of multi-layer GCNs, showing higher-degree nodes converge faster under standard GCN, and prove GCNII can express a K-th order polynomial filter with arbitrary coefficients (Theorem 2).
• Relate the model to iterative shrinkage-thresholding and ResNet-style architecture to justify initialization and skip connections.

Experimental results

Research questions

• RQ1 Can a GCN be made truly deep without succumbing to over-smoothing, by introducing simple architectural changes?
• RQ2 Do initial residual connections and identity mappings enable GCNs to express richer polynomial filters and maintain information from input features as depth grows?
• RQ3 What are the theoretical implications for stationarity, convergence, and dependency on node degree in deep GCN variants?
• RQ4 How does GCNII perform compared to state-of-the-art models across semi-supervised and full-supervised node classification datasets and varying depths?

Key findings

• GCNII achieves state-of-the-art results on Cora, Citeseer, and Pubmed for semi-supervised node classification (example results: GCNII 85.5±0.5 on Cora, 73.4±0.6 on Citeseer, 80.2±0.4 on Pubmed).
• GCNII* (a variant) attains comparable performance with slightly different parameterization.
• GCNII enables deep networks (up to 64 layers) with consistent improvements over shallow baselines and other deep models; e.g., 64-layer GCNII reaches 85.5 on Cora and 80.2 on Pubmed.
• Theoretical results show a K-layer GCN tends to converge to a stationary vector; GCNII can express a K-order polynomial filter with arbitrary coefficients, addressing expressive power limitations of vanilla GCN (Theorem 2).
• The stationary convergence rate in vanilla GCN is influenced by node degree, with higher-degree nodes more prone to over-smoothing (Conjecture 1 and Theorem 1 discussion).
• Empirical results indicate that deep GCNII consistently outperforms baselines across datasets and depths, and can surpass DropEdge and JKNet variants in deep regimes.