[Paper Review] Stochastic Training of Graph Convolutional Networks with Variance Reduction

Jianfei Chen, Jun Zhu+1 more|arXiv (Cornell University)|Oct 29, 2017

Advanced Graph Neural Networks9 references181 citations

TL;DR

The paper proposes preprocessing and control-variate techniques to enable fast, unbiased stochastic training of GCNs with neighbor sampling, achieving exact GCN-like performance with faster convergence.

ABSTRACT

Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes the representation of a node recursively from its neighbors, making the receptive field size grow exponentially with the number of layers. Previous attempts on reducing the receptive field size by subsampling neighbors do not have a convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop control variate based algorithms which allow sampling an arbitrarily small neighbor size. Furthermore, we prove new theoretical guarantee for our algorithms to converge to a local optimum of GCN. Empirical results show that our algorithms enjoy a similar convergence with the exact algorithm using only two neighbors per node. The runtime of our algorithms on a large Reddit dataset is only one seventh of previous neighbor sampling algorithms.

Motivation & Objective

• Motivate efficient semi-supervised node classification on graph-structured data using GCNs.
• Address the high computational cost of GCNs with stochastic training.
• Introduce preprocessing and control-variate techniques to reclaim fast convergence and exact local optima.
• Provide theoretical convergence guarantees for the proposed approach.

Proposed method

• Introduce preprocessing to reduce the GCN depth by one without changing the expected output.
• Apply neighbor sampling to approximate neighborhood aggregation and analyze its limitations.
• Propose a control-variate (CV) method that uses historical activations to create an unbiased gradient estimator.
• Develop a variance-reduction scheme (CVD) that also accounts for dropout and maintains mean activations.
• Provide theoretical guarantees: CV yields exact predictions and an unbiased gradient, with convergence under standard assumptions.
• Compare time complexity and convergence behavior across Exact, NS, NS+PP, CV+PP, and CVD+PP variants.

Experimental results

Research questions

• RQ1 Can we reduce the depth of GCNs via preprocessing without sacrificing discriminative performance?
• RQ2 Does neighbor sampling degrade performance, and can variance reduction recover exact GCN behavior?
• RQ3 Can a control-variate approach yield unbiased gradients and exact local optima comparable to full-batch GCN training?
• RQ4 What are the trade-offs in time complexity and convergence when combining preprocessing with variance-reduction techniques?
• RQ5 Do the proposed methods maintain performance under dropout and provide theoretical convergence guarantees?

Key findings

• Preprocessing plus control-variate approaches can match exact GCN performance while offering faster per-epoch time.
• Control-variate yields unbiased gradients and converges to the exact GCN local optimum under zero dropout.
• Variance-reduction techniques mitigate the instability introduced by neighbor sampling and dropout.
• The method provides theoretical guarantees for unbiased gradient and convergence under standard smoothness assumptions.
• Different algorithm variants demonstrate favorable trade-offs between accuracy and computational efficiency on benchmark datasets.