[Paper Review] PipeDream: Fast and Efficient Pipeline Parallel DNN Training

Aaron Harlap, Deepak Narayanan+5 more|arXiv (Cornell University)|Jun 8, 2018

Advanced Neural Network Applications4 references97 citations

TL;DR

PipeDream combines pipelined model parallelism with data parallelism to enable fast pipeline-parallel DNN training, significantly reducing communication and overlapping computation and communication to improve time-to-accuracy compared to data-parallel BSP.

ABSTRACT

PipeDream is a Deep Neural Network(DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize "time to target accuracy". Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.

Motivation & Objective

• Motivate the need for efficient training of large DNNs where data-parallelism suffers from high communication costs and bandwidth limits.
• Propose a pipeline-parallel training approach that combines pipelined model parallelism with data parallelism to balance computation and communication.
• Automatically partition DNNs into pipeline stages to minimize makespan and enable replication where beneficial.
• Ensure correct learning under pipelining by managing weight versions and synchronization across stages (weight stashing and vertical sync).
• Demonstrate empirical performance gains across CNNs and RNNs on real clusters using commodity hardware.

Proposed method

• Partition DNN layers into stages across GPUs to balance compute and minimize inter-stage communication.
• Profile a short run of the DNN to estimate per-layer compute time, activation sizes, and parameter sizes for partitioning.
• Use dynamic programming to find an optimal pipeline partitioning, including stage replication and NOAM (number of active minibatches).
• Employ a 1-forward-1-backward (1F1B) scheduling policy to keep all GPUs busy in steady state.
• Apply weight stashing to keep per-minibatch forward and backward passes consistent with a given weight version, and use vertical sync to align updates across stages.
• Allocate GPU memory upfront for minibatches and intermediate state to minimize allocation overhead.

Experimental results

Research questions

• RQ1 How can DNN training be parallelized beyond data parallelism to reduce communication and improve hardware efficiency on commodity GPUs?
• RQ2 Can automatic partitioning of layers into pipeline stages, with optional replication, minimize training time for large models?
• RQ3 What scheduling and consistency mechanisms (like weight stashing and vertical sync) are necessary to preserve convergence in pipeline-parallel training?
• RQ4 How does pipeline-parallel training perform in terms of time-to-accuracy compared with data-parallel BSP across CNN and RNN models?
• RQ5 What are the practical gains when using PipeDream on real clusters with varying models and hardware configurations?

Key findings

• PipeDream reduces inter-worker communication by up to 95% compared with data-parallel training for large DNNs.
• PipeDream overlaps communication and computation, achieving higher hardware efficiency than BSP.
• On five models (including Inception-v3, VGG16, ResNet-50, AlexNet, and S2VT), PipeDream speeds up time-to-accuracy up to 5.12x (varies by model).
• PipeDream demonstrates up to 6.76x speedup on AlexNet and up to 3x on S2VT relative to data-parallel BSP, across two clusters.
• The system supports both CNNs and RNNs (seq-to-seq) and works with commodity cloud GPUs.
• Experiments indicate PipeDream is up to 5x faster in time-to-accuracy than data-parallel training on the tested configurations.