[Paper Review] FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

Jie Chen, Tengfei Ma+1 more|arXiv (Cornell University)|Jan 30, 2018

Advanced Graph Neural Networks639 citations

TL;DR

FastGCN reframes graph convolutions as integral transforms, enabling Monte Carlo sampling with importance weighting to train inductively and with much lower per-batch cost than standard GCNs and GraphSAGE, while maintaining comparable accuracy.

ABSTRACT

The graph convolutional networks (GCN) recently proposed by Kipf and Welling are an effective graph model for semi-supervised learning. This model, however, was originally designed to be learned with the presence of both training and test data. Moreover, the recursive neighborhood expansion across layers poses time and memory challenges for training with large, dense graphs. To relax the requirement of simultaneous availability of test data, we interpret graph convolutions as integral transforms of embedding functions under probability measures. Such an interpretation allows for the use of Monte Carlo approaches to consistently estimate the integrals, which in turn leads to a batched training scheme as we propose in this work---FastGCN. Enhanced with importance sampling, FastGCN not only is efficient for training but also generalizes well for inference. We show a comprehensive set of experiments to demonstrate its effectiveness compared with GCN and related models. In particular, training is orders of magnitude more efficient while predictions remain comparably accurate.

Motivation & Objective

• Introduce an inductive learning framework for graph convolutions that removes dependence on test data.
• Reduce training time and memory load for GCNs on large, dense graphs via sampling.
• Provide a principled Monte Carlo estimator for layer-wise convolutions and loss.
• Develop variance-reduction techniques to improve sampling efficiency while preserving accuracy.

Proposed method

• Interpret graph convolutions as integral transforms of vertex embedding functions under a probability measure.
• Use Monte Carlo sampling to estimate layer transforms and the loss, enabling batched training.
• Apply importance sampling to reduce estimator variance and derive an optimal sampling distribution.
• Provide a batched training algorithm (FastGCN) with sampling per layer and backpropagation of batch gradients.
• Propose two variants: uniform sampling and importance sampling; compute precomputation of AH(0) to speed training.
• Show how the method reduces memory footprint by sampling vertices rather than entire neighborhoods.

Experimental results

Research questions

• RQ1 Can graph convolutions be reformulated as integral transforms to enable inductive learning without requiring test data?
• RQ2 Does vertex-wise sampling (as opposed to neighborhood sampling) yield substantial computational savings for GCNs on large graphs?
• RQ3 Can importance sampling improve the variance of Monte Carlo estimates in FastGCN, compared to uniform sampling?
• RQ4 How does FastGCN perform in terms of training time and predictive accuracy compared with GCN and GraphSAGE on standard benchmarks?
• RQ5 What are the practical implications for inference when using an inductive FastGCN framework?

Key findings

• FastGCN achieves order-of-magnitude speedups over GCN and GraphSAGE in per-batch training time.
• Importance sampling consistently yields higher accuracy than uniform sampling in experiments.
• Prediction accuracies with FastGCN are highly comparable to those of GCN and GraphSAGE across datasets.
• Precomputing AH(0) for the input layer can substantially reduce training time with comparable accuracy.
• FastGCN enables inductive learning by separating training/test data and avoiding full neighborhood expansion.