[Paper Review] Adaptive Sampling Towards Fast Graph Representation Learning

Wenbing Huang, Tong Zhang+2 more|arXiv (Cornell University)|Sep 14, 2018

Advanced Graph Neural NetworksComputer Science228 citations

TL;DR

The paper introduces layer-wise adaptive sampling with variance-reducing training and skip connections to accelerate GCNs on large graphs, achieving faster convergence and improved accuracy over baselines.

ABSTRACT

Graph Convolutional Networks (GCNs) have become a crucial tool on learning representations of graph vertices. The main challenge of adapting GCNs on large-scale graphs is the scalability issue that it incurs heavy cost both in computation and memory due to the uncontrollable neighborhood expansion across layers. In this paper, we accelerate the training of GCNs through developing an adaptive layer-wise sampling method. By constructing the network layer by layer in a top-down passway, we sample the lower layer conditioned on the top one, where the sampled neighborhoods are shared by different parent nodes and the over expansion is avoided owing to the fixed-size sampling. More importantly, the proposed sampler is adaptive and applicable for explicit variance reduction, which in turn enhances the training of our method. Furthermore, we propose a novel and economical approach to promote the message passing over distant nodes by applying skip connections. Intensive experiments on several benchmarks verify the effectiveness of our method regarding the classification accuracy while enjoying faster convergence speed.

Motivation & Objective

• Motivate and address the scalability challenge of Graph Convolutional Networks (GCNs) on large graphs due to neighborhood expansion.
• Propose a layer-wise sampling framework that shares sampled neighborhoods across parents to fix layer sizes and control expansion.
• Introduce an adaptive sampler that minimizes variance in a tractable way and integrates a variance-reducing objective into training.
• Enhance long-range message passing via skip connections to preserve second-order proximity without heavy extra computations.

Proposed method

• Reformulate GCN updates as expectations and replace full neighborhood expansion with Monte-Carlo estimates.
• Develop layer-wise sampling where sampling is performed once per layer and neighborhoods are shared among nodes in the upper layer.
• Design an adaptive sampler q(u_j) derived from a self-dependent function g(x(u_j)) to approximate the variance-minimizing distribution (Eq. 9) and integrate variance reduction into a hybrid loss.
• Introduce a skip-connection scheme that reuses nodes from the (l-1)-th layer to enable 2-hop neighborhoods without computing A^2 directly (Eq. 12–13).
• Relate the sampler to existing methods (GraphSAGE, FastGCN) and discuss attention-inspired variants (GAT-like) adapted to the layer-wise framework.
• Provide an inductive learning setting and empirical evaluation on standard graphs (Cora, Citeseer, Pubmed, Reddit).

Experimental results

Research questions

• RQ1 Can layer-wise sampling with shared neighborhoods speed up training of GCNs while maintaining or improving accuracy on standard graph benchmarks?
• RQ2 Does an adaptive, variance-reducing sampler outperform node-wise or IID layer sampling in terms of stability and convergence?
• RQ3 Does incorporating skip connections to preserve second-order proximity improve convergence and predictive performance?
• RQ4 How does the proposed method compare to existing sampling-based or attention-based graph models on benchmark datasets?

Key findings

• Adapt achieves higher test accuracy than strong baselines on Cora, Citeseer, Pubmed, and Reddit (e.g., 0.8744 on Cora, 0.7966 on Citeseer, 0.9060 on Pubmed, 0.9627 on Reddit).
• Adapt outperforms Full GCN, IID, GraphSAGE, and FastGCN baselines in reported results, with notably faster convergence (training time per epoch) and improved stability.
• Variance reduction via the adaptive sampler (lambda > 0) yields better performance than removing variance term (lambda = 0) on Cora and Reddit; Citeseer shows smaller variance impact.
• Skip connections speed up convergence significantly (e.g., reducing convergence epochs from ~150 to ~100 on Cora) with modest changes in final accuracy.
• An explicit 2-hop sampling variant (using A^2) can further boost accuracy, though skip connections offer a more computation-friendly alternative for large graphs.
• Compared to IID and node-wise sampling, layer-wise sampling with conditional dependence captures between-layer correlations and yields faster, more stable training.