[Paper Review] Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training

Saptadeep Pal, Eiman Ebrahimi+6 more|arXiv (Cornell University)|Jul 30, 2019

Advanced Neural Network Applications20 references72 citations

TL;DR

This paper proposes a hybrid multi-GPU training strategy that combines data parallelism (DP) with model parallelism (MP) to overcome DP's scalability and statistical efficiency limitations at scale. By using an analytical framework and DLPlacer—a mixed-integer linear programming tool for optimal operation-to-device placement—it demonstrates that hybrid training achieves 26.5%, 8%, and 22% end-to-end training speedups for Inception-V3, GNMT, and BigLSTM respectively, outperforming DP alone.

ABSTRACT

Deploying deep learning (DL) models across multiple compute devices to train large and complex models continues to grow in importance because of the demand for faster and more frequent training. Data parallelism (DP) is the most widely used parallelization strategy, but as the number of devices in data parallel training grows, so does the communication overhead between devices. Additionally, a larger aggregate batch size per step leads to statistical efficiency loss, i.e., a larger number of epochs are required to converge to a desired accuracy. These factors affect overall training time and beyond a certain number of devices, the speedup from leveraging DP begins to scale poorly. In addition to DP, each training step can be accelerated by exploiting model parallelism (MP). This work explores hybrid parallelization, where each data parallel worker is comprised of more than one device, across which the model dataflow graph (DFG) is split using MP. We show that at scale, hybrid training will be more effective at minimizing end-to-end training time than exploiting DP alone. We project that for Inception-V3, GNMT, and BigLSTM, the hybrid strategy provides an end-to-end training speedup of at least 26.5%, 8%, and 22% respectively compared to what DP alone can achieve at scale.

Motivation & Objective

• To address the diminishing returns in data parallel training as GPU count increases, particularly due to rising communication overhead and reduced statistical efficiency.
• To identify the optimal point at which model parallelism should be introduced to maintain training speedup beyond the scalability limits of pure data parallelism.
• To develop a systematic framework for determining the best parallelization strategy (DP vs. hybrid) based on model architecture and hardware constraints.
• To create DLPlacer, an integer linear programming-based tool for optimal operation-to-device placement that maximizes model parallelism speedup.
• To empirically validate that hybrid parallelization significantly reduces end-to-end training time across diverse deep learning models.

Proposed method

• The authors develop an analytical model to predict the cross-over point where data parallelism's performance degradation exceeds the speedup achievable via model parallelism.
• They implement 2-way model parallel versions of Inception-V3, GNMT, and BigLSTM using pipelining and inter-layer partitioning to enable concurrent execution across multiple GPUs.
• DLPlacer is introduced as an integer linear programming formulation to find the optimal placement of operations across devices, minimizing communication overhead and maximizing parallelism.
• The framework evaluates training time and convergence across different global batch sizes and device counts, comparing pure DP with hybrid DP+MP strategies.
• The authors use real-world models and hardware configurations to project end-to-end training speedups, validating results with measured speedups on Inception-V3.

Experimental results

Research questions

• RQ1 At what scale does data parallelism alone begin to lose training speedup due to communication overhead and reduced statistical efficiency?
• RQ2 How can model parallelism be effectively combined with data parallelism to maintain or improve training throughput at scale?
• RQ3 What is the optimal device placement strategy for model parallelism that maximizes speedup while minimizing communication costs?
• RQ4 Can a systematic framework predict the best parallelization strategy (DP vs. hybrid) for a given model and hardware setup?
• RQ5 To what extent can hybrid parallelization reduce end-to-end training time compared to data parallelism alone across diverse deep learning architectures?

Key findings

• For Inception-V3, the hybrid strategy achieves a projected end-to-end training speedup of at least 26.5% compared to data parallelism alone at scale.
• For GNMT, the hybrid approach provides a minimum of 8% end-to-end training speedup over pure data parallelism.
• For BigLSTM, the hybrid strategy delivers a minimum 22% reduction in end-to-end training time compared to data parallelism.
• DLPlacer’s predicted model parallel speedup for Inception-V3 (1.32x) is within 6% of the actual measured speedup, validating its accuracy in placement optimization.
• The study identifies a unique cross-over point for each model where data parallelism’s diminishing returns are overcome by model parallelism’s speedup, enabling optimal hybrid strategy selection.
• The results demonstrate that hybrid parallelization is essential for scaling deep learning training beyond a certain number of GPUs, where pure data parallelism becomes inefficient.