[Paper Review] Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication

Meng Zhang, Qinghao Hu+3 more|arXiv (Cornell University)|Mar 2, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes Sylvie, a distributed GNN training framework that uses asynchronous one-bit communication to drastically reduce inter-GPU communication overhead in full-graph GNN training. By applying low-bit quantization and dequantization with error compensation, Sylvie achieves up to 28.1× speedup over baseline methods while preserving model accuracy.

ABSTRACT

Training Graph Neural Networks (GNNs) on large graphs is challenging due to the conflict between the high memory demand and limited GPU memory. Recently, distributed full-graph GNN training has been widely adopted to tackle this problem. However, the substantial inter-GPU communication overhead can cause severe throughput degradation. Existing communication compression techniques mainly focus on traditional DNN training, whose bottleneck lies in synchronizing gradients and parameters. We find they do not work well in distributed GNN training as the barrier is the layer-wise communication of features during the forward pass & feature gradients during the backward pass. To this end, we propose an efficient distributed GNN training framework Sylvie, which employs one-bit quantization technique in GNNs and further pipelines the curtailed communication with computation to enormously shrink the overhead while maintaining the model quality. In detail, Sylvie provides a lightweight Low-bit Module to quantize the sent data and dequantize the received data back to full precision values in each layer. Additionally, we propose a Bounded Staleness Adaptor to control the introduced staleness to achieve further performance enhancement. We conduct theoretical convergence analysis and extensive experiments on various models & datasets to demonstrate Sylvie can considerably boost the training throughput by up to 28.1x.

Motivation & Objective

• Address the high inter-GPU communication overhead in distributed full-graph GNN training, which limits scalability and throughput.
• Overcome the limitations of existing compression techniques, which are optimized for DNNs and not effective for GNNs due to layer-wise feature and gradient communication.
• Maintain high model accuracy while drastically reducing communication costs through low-bit quantization and asynchronous pipeline execution.
• Design a system that generalizes across diverse GNN models and datasets without sacrificing convergence or performance.

Proposed method

• Introduce a Low-bit Module that quantizes features and gradients to 1-bit for transmission and dequantizes them back to full precision at the receiver using error-compensated reconstruction.
• Implement an asynchronous pipeline (Sylvie-A) that overlaps communication with computation to hide latency and improve resource utilization.
• Propose a Bounded Staleness Adaptor to control the impact of stale features and gradients in asynchronous training, improving convergence stability.
• Design a Graph Engine to partition the graph across GPUs and identify communication-critical nodes to minimize redundant data transfer.
• Use a trainer module that performs forward and backward passes using dequantized data, ensuring model consistency despite quantization.
• Apply quantization selectively to only the communicated tensors, preserving full-precision computation in the model core.

Experimental results

Research questions

• RQ1 Can one-bit quantization effectively reduce communication overhead in distributed GNN training without degrading model accuracy?
• RQ2 How does asynchronous pipeline execution with low-bit communication impact training throughput and convergence in GNNs?
• RQ3 To what extent does staleness in features and gradients affect model convergence, and can it be bounded to maintain performance?
• RQ4 Can the proposed framework generalize across different GNN architectures and real-world graph datasets?
• RQ5 What is the trade-off between communication reduction, computation overhead, and memory footprint in low-bit GNN training?

Key findings

• Sylvie reduces communication overhead by 89.8% and training time per epoch by 84.2% compared to FP32 baseline on the Reddit dataset.
• Using 1-bit quantization, Sylvie achieves up to 28.1× speedup in training throughput on two-server setups, as shown in Figure 1.
• Quantizing all embeddings and gradients to 1-bit causes severe accuracy drops (e.g., 70.6% on Ogbn-products vs. 78.86% in Sylvie-S), proving selective quantization is essential.
• The Low-bit Module contributes only 5.9% of total training time on a two-server setup, indicating negligible computational overhead.
• The Bounded Staleness Adaptor improves convergence in scenarios where asynchronous updates would otherwise degrade performance.
• The framework maintains high accuracy across multiple models (GraphSAGE, GCN, GAT) and datasets (Reddit, Yelp, Ogbn-products), with minimal accuracy loss compared to full-precision training.