[Paper Review] A Comprehensive Survey on Distributed Training of Graph Neural Networks

Haiyang Lin, Mingyu Yan+5 more|arXiv (Cornell University)|Nov 10, 2022

Advanced Graph Neural Networks4 citations

TL;DR

This paper provides a comprehensive survey of distributed training for Graph Neural Networks (GNNs), categorizing workflows, computational and communication patterns, and optimization techniques. It highlights key challenges like over-smoothing and communication overhead, and identifies opportunities in deep GNNs and model parallelism for large-scale graph learning.

ABSTRACT

Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training which distributes the workload of training across multiple computing nodes. At present, the volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication. Moreover, the approaches reported in these studies exhibit significant divergence. This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training. As a result, there is a pressing need for a survey to provide correct recognition, analysis, and comparisons in this field. In this paper, we provide a comprehensive survey of distributed GNN training by investigating various optimization techniques used in distributed GNN training. First, distributed GNN training is classified into several categories according to their workflows. In addition, their computational patterns and communication patterns, as well as the optimization techniques proposed by recent work are introduced. Second, the software frameworks and hardware platforms of distributed GNN training are also introduced for a deeper understanding. Third, distributed GNN training is compared with distributed training of deep neural networks, emphasizing the uniqueness of distributed GNN training. Finally, interesting issues and opportunities in this field are discussed.

Motivation & Objective

• To address the growing complexity and fragmentation in distributed GNN training research, which hinders newcomers' understanding of workflows, patterns, and optimization strategies.
• To systematically classify distributed GNN training methods based on execution workflows, including full-batch and mini-batch training paradigms.
• To analyze computational and communication patterns across different distributed training approaches, emphasizing performance bottlenecks like memory and communication overhead.
• To contrast distributed GNN training with distributed DNN training, highlighting unique challenges arising from graph topology and message passing.
• To identify open issues and future research opportunities, particularly in enabling deep GNN architectures and scalable model parallelism.

Proposed method

• The paper classifies distributed GNN training into dispatch-workload-based and preset-workload-based full-batch training, and individual-sample-based and joint-sample-based mini-batch training.
• It analyzes computational patterns such as aggregation and combination operations in GNNs, and communication patterns involving feature and gradient exchanges across nodes.
• Optimization techniques are reviewed, including activation rematerialization, mini-batch pipelining, and data caching to reduce communication and memory costs.
• The survey evaluates software frameworks and hardware platforms used in distributed GNN training, assessing their impact on system efficiency.
• It draws comparisons with distributed DNN training, emphasizing differences due to graph-specific data dependencies and non-Euclidean structures.
• The paper discusses emerging trends, such as deeper GNN models and the potential of pipeline and model parallelism, to overcome performance degradation issues.

Experimental results

Research questions

• RQ1 How can distributed GNN training be systematically classified based on its workflow and execution model?
• RQ2 What are the dominant computational and communication patterns in distributed GNN training, and how do they differ from those in distributed DNN training?
• RQ3 What optimization techniques effectively reduce communication and memory overhead in distributed GNN training?
• RQ4 How do performance degradation issues like over-smoothing and under-reaching affect distributed training strategies?
• RQ5 What future research directions are most promising for scaling GNNs on large-scale graphs?

Key findings

• Distributed mini-batch training is currently dominant due to its similarity to data parallelism in DNNs, despite challenges in model partitioning.
• Performance degradation in GNNs—particularly over-smoothing and under-reaching—limits model depth, typically restricting models to 2–4 layers.
• Techniques like DropEdge and DropNode mitigate over-smoothing, while virtual edges and super vertices help address long-range dependency issues.
• Communication and memory overhead remain major bottlenecks, driving research into activation rematerialization, pipelined transmission, and data caching.
• The rise of deeper GNN models is expected to favor model parallelism and pipeline parallelism, shifting optimization focus from data to model partitioning.
• Despite rapid progress, significant research gaps remain in scalable, efficient, and generalizable distributed training frameworks for large-scale graphs.