[Paper Review] Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

Difan Zou, Ziniu Hu+4 more|arXiv (Cornell University)|Nov 17, 2019

Advanced Graph Neural Networks87 citations

TL;DR

LADIES introduces layer-dependent importance sampling to train deep and large GCNs with reduced memory/time costs and improved generalization compared to prior sampling methods.

ABSTRACT

Graph convolutional networks (GCNs) have recently received wide attentions, due to their successful applications in different graph tasks and different domains. Training GCNs for a large graph, however, is still a challenge. Original full-batch GCN training requires calculating the representation of all the nodes in the graph per GCN layer, which brings in high computation and memory costs. To alleviate this issue, several sampling-based methods have been proposed to train GCNs on a subset of nodes. Among them, the node-wise neighbor-sampling method recursively samples a fixed number of neighbor nodes, and thus its computation cost suffers from exponential growing neighbor size; while the layer-wise importance-sampling method discards the neighbor-dependent constraints, and thus the nodes sampled across layer suffer from sparse connection problem. To deal with the above two problems, we propose a new effective sampling algorithm called LAyer-Dependent ImportancE Sampling (LADIES). Based on the sampled nodes in the upper layer, LADIES selects their neighborhood nodes, constructs a bipartite subgraph and computes the importance probability accordingly. Then, it samples a fixed number of nodes by the calculated probability, and recursively conducts such procedure per layer to construct the whole computation graph. We prove theoretically and experimentally, that our proposed sampling algorithm outperforms the previous sampling methods in terms of both time and memory costs. Furthermore, LADIES is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.

Motivation & Objective

• Motivate training deep GCNs on large graphs given full-batch costs and redundancy in node-wise sampling.
• Develop a layer-dependent sampling scheme to maintain connectivity and reduce variance.
• Prove theoretical efficiency and variance benefits over existing methods.
• Demonstrate empirical gains in running time, memory, and accuracy on benchmark datasets.

Proposed method

• Propose LADIES that, for each layer, builds a bipartite subgraph from the upper-layer sampled nodes and their neighbors.
• Compute layer-wise importance probabilities using p_i^{(l-1)} = ||Q^{(l)} P_{*,i}||_2^2 / ||Q^{(l)} P||_F^2 to guide sampling.
• Sample a fixed number of nodes per layer based on the calculated probabilities and construct a dense, normalized sampled adjacency tilde{P}^{(l-1)} to propagate embeddings.
• Use top-down, layer-dependent sampling to ensure connectivity and avoid exponential receptive field growth.
• Normalize tilde{P}^{(l)} by row-sums to stabilize training.
• Provide theoretical analysis of memory/time complexity and variance, and empirical validation on multiple datasets.

Experimental results

Research questions

• RQ1 How can layer-dependent sampling improve the connectivity and efficiency of the computation graph in deep GCNs?
• RQ2 Can LADIES provide lower memory/time complexity and reduced variance compared to node-wise and layer-wise prior methods?
• RQ3 Does LADIES improve or preserve predictive accuracy and generalization on standard graph benchmarks?
• RQ4 What sample sizes suffice to achieve strong performance on very large graphs?

Key findings

• LADIES achieves lower memory and time costs than node-wise sampling methods and maintains comparable or better accuracy.
• Compared to FastGCN, LADIES has strictly better variance due to a smaller effective connected node set and benefits from small sample sizes on large graphs.
• On benchmarks (Cora, Citeseer, Pubmed, Reddit), LADIES attains the best test accuracy with smaller samples (e.g., 64) and deep architectures.
• LADIES demonstrates strong generalization, often outperforming full-batch GCN on validation/test despite using stochastic sampling.
• LADIES scales to very large graphs and deep GCNs without exponential growth in computation.