[Paper Review] Residual Gated Graph ConvNets

Xavier Bresson, Thomas Laurent|arXiv (Cornell University)|Nov 20, 2017

Advanced Graph Neural Networks25 references246 citations

TL;DR

The paper rigorously compares graph RNNs and graph ConvNets on variable-length graphs, proposes residual gated graph ConvNets, and shows ConvNets outperform RNNs in accuracy and speed, with residuals providing additional gains.

ABSTRACT

Graph-structured data such as social networks, functional brain networks, gene regulatory networks, communications networks have brought the interest in generalizing deep learning techniques to graph domains. In this paper, we are interested to design neural networks for graphs with variable length in order to solve learning problems such as vertex classification, graph classification, graph regression, and graph generative tasks. Most existing works have focused on recurrent neural networks (RNNs) to learn meaningful representations of graphs, and more recently new convolutional neural networks (ConvNets) have been introduced. In this work, we want to compare rigorously these two fundamental families of architectures to solve graph learning tasks. We review existing graph RNN and ConvNet architectures, and propose natural extension of LSTM and ConvNet to graphs with arbitrary size. Then, we design a set of analytically controlled experiments on two basic graph problems, i.e. subgraph matching and graph clustering, to test the different architectures. Numerical results show that the proposed graph ConvNets are 3-17% more accurate and 1.5-4x faster than graph RNNs. Graph ConvNets are also 36% more accurate than variational (non-learning) techniques. Finally, the most effective graph ConvNet architecture uses gated edges and residuality. Residuality plays an essential role to learn multi-layer architectures as they provide a 10% gain of performance.

Motivation & Objective

• Motivate the need for neural networks that operate on graphs with variable length across domains (social, brain, regulatory, etc.).
• Compare graph recurrent networks (RNNs) and graph convolutional networks (ConvNets) and extend them to arbitrary graph sizes.
• Design and evaluate residual gated graph ConvNets to improve performance and training of deep graph models.

Proposed method

• Review existing graph RNN and ConvNet architectures.
• Propose a multi-layer graph LSTM and gated graph ConvNet with edge gating.
• Incorporate residual connections between layers to enable deeper architectures.
• Use analytically controlled experiments on subgraph matching and graph clustering to compare models.
• Adopt Batch Normalization and standard optimization schedules to ensure fair comparison.

Experimental results

Research questions

• RQ1 Which architecture—graph RNNs or graph ConvNets—better handles graphs with arbitrary size and depth?
• RQ2 Do gated edges and residual connections improve learning on graph-structured tasks?
• RQ3 How do different architectures perform on subgraph matching and graph clustering under varying noise and budget constraints?

Key findings

• Graph ConvNets achieve 3-17% higher accuracy and 1.5-4x faster training than graph RNNs on the tested tasks.
• Graph ConvNets are 36% more accurate than variational (non-learning) techniques on the same tasks.
• The most effective graph ConvNet uses gated edges and residual connections.
• Residuality provides a roughly 10% gain in performance when stacking more than 6 layers.
• ConvNet architectures benefit from increased depth, while RNN-based models can degrade with too many layers.