[Paper Review] DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

Yu Rong, Wenbing Huang+2 more|arXiv (Cornell University)|Jul 25, 2019

Advanced Graph Neural Networks33 references627 citations

TL;DR

DropEdge randomly drops edges during training to augment graph data and reduce message-passing density, enabling deeper GCNs and improved node classification performance across multiple backbones.

ABSTRACT

\emph{Over-fitting} and \emph{over-smoothing} are two main obstacles of developing deep Graph Convolutional Networks (GCNs) for node classification. In particular, over-fitting weakens the generalization ability on small dataset, while over-smoothing impedes model training by isolating output representations from the input features with the increase in network depth. This paper proposes DropEdge, a novel and flexible technique to alleviate both issues. At its core, DropEdge randomly removes a certain number of edges from the input graph at each training epoch, acting like a data augmenter and also a message passing reducer. Furthermore, we theoretically demonstrate that DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by it. More importantly, our DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance. Extensive experiments on several benchmarks verify that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs. The effect of DropEdge on preventing over-smoothing is empirically visualized and validated as well. Codes are released on~\url{https://github.com/DropEdge/DropEdge}.

Motivation & Objective

• Motivate the challenges of over-fitting and over-smoothing in deep GCNs for node classification.
• Propose DropEdge as a flexible edge-dropping data augmentation technique.
• Analyze how DropEdge slows over-smoothing and preserves information in deep GCNs.

Proposed method

• DropEdge randomly removes a fraction p of edges from the input graph during each training epoch to create perturbed adjacency matrices.
• Renormalize the dropped adjacency as in Kipf & Welling and use it in forward propagation.
• Allow a layer-wise variant where different layers use independently dropped adjacency matrices.
• Provide theoretical justification showing DropEdge slows convergence of over-smoothing or reduces information loss.
• Show compatibility and empirical gains across backbones like GCN, ResGCN, JKNet, IncepGCN, GraphSAGE.

Experimental results

Research questions

• RQ1 Can DropEdge alleviate over-fitting in deep GCNs on small graphs while maintaining representation quality?
• RQ2 Does DropEdge slow down over-smoothing to enable deeper GCNs without losing input feature information?
• RQ3 How well does DropEdge transfer across different backbone architectures for node classification?
• RQ4 What is the empirical impact of DropEdge on learning dynamics and validation performance across benchmarks?

Key findings

• DropEdge consistently improves testing accuracy for various backbones across multiple datasets.
• DropEdge enables deeper architectures (beyond 2 layers) to perform better and even avoid memory issues on some settings.
• The method reduces over-fitting as shown by lower validation loss on Cora with 4-layer models.
• DropEdge can be combined with Dropout for additional gains.
• Layer-wise DropEdge offers marginal training improvements at higher computational cost.
• DropEdge outperforms several SOTA methods on Reddit and other datasets in reported experiments.