[Paper Review] PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication

Cheng Wan, Youjie Li+4 more|arXiv (Cornell University)|Mar 20, 2022

Advanced Graph Neural Networks4 citations

TL;DR

PipeGCN proposes a pipelined communication-computation scheme to hide inter-partition communication overhead in distributed Graph Convolutional Network (GCN) training, enabling efficient full-graph training. It achieves near-vanilla convergence rates despite feature and gradient staleness, with throughput improvements of 1.7× to 28.5× and consistent accuracy across multiple datasets and GPU configurations.

ABSTRACT

Graph Convolutional Networks (GCNs) is the state-of-the-art method for learning graph-structured data, and training large-scale GCNs requires distributed training across multiple accelerators such that each accelerator is able to hold a partitioned subgraph. However, distributed GCN training incurs prohibitive overhead of communicating node features and feature gradients among partitions for every GCN layer during each training iteration, limiting the achievable training efficiency and model scalability. To this end, we propose PipeGCN, a simple yet effective scheme that hides the communication overhead by pipelining inter-partition communication with intra-partition computation. It is non-trivial to pipeline for efficient GCN training, as communicated node features/gradients will become stale and thus can harm the convergence, negating the pipeline benefit. Notably, little is known regarding the convergence rate of GCN training with both stale features and stale feature gradients. This work not only provides a theoretical convergence analysis but also finds the convergence rate of PipeGCN to be close to that of the vanilla distributed GCN training without any staleness. Furthermore, we develop a smoothing method to further improve PipeGCN's convergence. Extensive experiments show that PipeGCN can largely boost the training throughput (1.7x~28.5x) while achieving the same accuracy as its vanilla counterpart and existing full-graph training methods. The code is available at https://github.com/RICE-EIC/PipeGCN.

Motivation & Objective

• To address the high communication overhead and synchronization cost in distributed GCN training across multiple accelerators.
• To enable efficient full-graph training without sacrificing model accuracy by minimizing communication bottlenecks.
• To theoretically analyze and mitigate convergence degradation caused by stale features and gradients in pipelined training.
• To develop a practical, low-overhead smoothing method that further improves convergence in the presence of staleness.
• To demonstrate significant throughput gains over vanilla distributed GCN and state-of-the-art full-graph training methods.

Proposed method

• Pipelining inter-partition communication with intra-partition computation to hide communication latency across GCN layers.
• Using a second CUDA stream to overlap communication with computation, batching all communication operations for efficiency.
• Introducing a smoothing method to reduce error from feature and gradient staleness, improving convergence stability.
• Applying dropout after communication to preserve gradient consistency and avoid noise in backpropagation.
• Maintaining model accuracy by ensuring consistent dropout masks across forward and backward passes in the pipelined setting.
• Scaling the method across multiple GPU servers with 10Gbps Ethernet, demonstrating robustness under diverse partitioning and hardware configurations.

Experimental results

Research questions

• RQ1 Can pipelining inter-partition communication with intra-partition computation significantly reduce training time in distributed GCN training without degrading model accuracy?
• RQ2 What is the theoretical convergence rate of GCN training when both features and feature gradients are stale due to pipelining?
• RQ3 How does the proposed smoothing method reduce the error introduced by staleness in feature and gradient communication?
• RQ4 How does PipeGCN compare in throughput and accuracy to vanilla distributed GCN and existing full-graph training methods across different datasets and hardware setups?
• RQ5 Can PipeGCN maintain high training efficiency and accuracy when scaling across multiple GPU servers with heterogeneous partitioning and network constraints?

Key findings

• PipeGCN achieves a 1.7× to 28.5× improvement in training throughput compared to vanilla distributed GCN training, with consistent accuracy across all evaluated datasets.
• The theoretical convergence rate of PipeGCN is O(T^(-2/3)), which is close to the O(T^(-1)) rate of vanilla distributed GCN training without staleness.
• The proposed smoothing method further improves convergence, reducing the error from staleness and enabling stable training with high throughput.
• On the Reddit dataset, PipeGCN maintains 97.0% to 97.14% accuracy across 2 to 16 GPU partitions, with speedups ranging from 1.16× to 1.65× on smaller setups and up to 1.65× on 3×4 GPU configurations.
• The method scales effectively across multiple GPU servers, maintaining accuracy and achieving 15% to 66% speedup regardless of partitioning or hardware configuration.
• Empirical results show that PipeGCN variants (e.g., PipeGCN-GF) achieve similar convergence speed to vanilla training but in half the wall-clock time, confirming the effectiveness of pipelining and smoothing.