[Paper Review] DeepGCNs: Can GCNs Go as Deep as CNNs?

Guohao Li, Matthias Müller+2 more|arXiv (Cornell University)|Apr 7, 2019

Advanced Graph Neural Networks49 references107 citations

TL;DR

Proposes residual, dense, and dilated connection strategies to train very deep Graph Convolutional Networks (GCNs) for point cloud semantic segmentation, achieving a 3.7% mIoU gain over state-of-the-art with a 56-layer GCN.

ABSTRACT

Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-the-art GCN models are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We do this by borrowing concepts from CNNs, specifically residual/dense connections and dilated convolutions, and adapting them to GCN architectures. Extensive experiments show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation. We believe that the community can greatly benefit from this work, as it opens up many opportunities for advancing GCN-based research.

Motivation & Objective

• Motivate and address the depth limitation of GCNs for non-Euclidean data.
• Adapt CNN-inspired mechanisms (residual/dense connections, dilated convolutions) to GCNs.
• Investigate the impact of these mechanisms on training stability and performance.
• Demonstrate deep GCN effectiveness on large-scale point cloud semantic segmentation (S3DIS).
• Provide comprehensive ablations to guide future deep GCN design.

Proposed method

• Represent graphs with feature-rich vertex embeddings and neighborhood aggregation.
• Introduce residual and dense connection schemes to improve gradient flow in deep GCNs (ResGCN, DenseGCN).
• Incorporate dilated k-NN based neighbor sampling to widen receptive fields without resolution loss.
• Adopt a three-block architecture: GCN Backbone Block, Fusion Block, and MLP Prediction Block.
• Use dynamic edges with dilated k-NN that are recomputed per layer during training (stochastic during training).
• Conduct extensive ablation studies on depth, width, neighbors, and dilation to isolate effects on stability and performance.

Experimental results

Research questions

• RQ1 Can CNN-inspired depth-enhancement techniques (residual/dense connections, dilation) be effectively translated to GCNs?
• RQ2 What is the impact of residual vs. dense connections and dilated aggregations on stability and performance of deep GCNs?
• RQ3 How deep can GCNs go for point cloud semantic segmentation, and what configurations yield the best gains?
• RQ4 Does dilated aggregation increase receptive field and improve segmentation accuracy without excessive over-smoothing?
• RQ5 What is the comparative performance of deep ResGCN/DenseGCN against state-of-the-art methods on S3DIS?

Key findings

• Residual graph connections are essential to stable training of deep GCNs; removing them drastically degrades performance.
• Dilated graph convolutions provide a notable gain in mIoU by expanding the receptive field, especially when combined with residual connections.
• Dynamic (recomputed) edges via dilated k-NN improve performance but incur higher computational cost.
• Dense connections yield similar gains to residuals but memory costs can be prohibitive, making residuals more practical in many settings.
• The reference model ResGCN-28 (28 layers, residual connections, dilated convolutions) achieves 60.0 mIoU on S3DIS, outperforming state-of-the-art methods (e.g., DGCNN) by noticeable margins.
• Deep architectures (56 layers) with these mechanisms achieve strong results, with ResGCN-56 attaining 53.64 OA and 53.64? actually 53.64 OA and - need to be precise; main takeaway is substantial mIoU improvements over baselines.