[Paper Review] HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

Jay Park, Gyeongchan Yun+6 more|arXiv (Cornell University)|May 28, 2020

Advanced Neural Network Applications41 citations

TL;DR

HetPipe integrates pipelined model parallelism with data parallelism to train large DNNs on heterogeneous GPUs, including weaker ones, achieving up to 49% faster convergence than state-of-the-art DP in experiments.

ABSTRACT

Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.

Motivation & Objective

• Motivate training of large DNNs on heterogeneous GPU clusters with mixed (potentially whimpy) GPUs.
• Enable efficient utilization by combining PMP within virtual workers and DP across virtual workers.
• Provide a convergence-guaranteed synchronization model suitable for heterogeneous, pipelined training.

Proposed method

• Introduce virtual workers composed of multiple GPUs to enable data parallelism under heterogeneity.
• Partition a DNN model into k partitions for PMP within each virtual worker to form a pipeline.
• Propose Wave Synchronous Parallel (WSP) as a convergence-guaranteed synchronization model that aggregates updates per wave.
• Use parameter servers for global weight synchronization with bounded global staleness.
• Provide a convergence proof for WSP.
• Modify TensorFlow to implement HetPipe and evaluate on a four-GPU heterogeneous cluster.

Experimental results

Research questions

• RQ1 Can large DNN models be trained efficiently on a heterogeneous GPU cluster by combining PMP and DP?
• RQ2 How should GPU resources be allocated and partitioned to maximize pipelined performance in HetPipe?
• RQ3 Does Wave Synchronous Parallel ensure convergence under heterogeneity and pipelined execution?
• RQ4 What performance gains are achieved over state-of-the-art DP approaches like Horovod with AllReduce?
• RQ5 How does HetPipe handle global and local staleness in a heterogeneous DP+PMP setting?

Key findings

• HetPipe achieves faster convergence than Horovod-based DP by up to 49% for VGG-19 and 39% for ResNet-152 in their heterogeneous cluster setup.
• HetPipe enables training of large models that could be too big for a single whimpy GPU by forming virtual workers.
• PMP within virtual workers and DP across virtual workers improve utilization of heterogeneous GPUs.
• WSP provides a convergence guarantee for the combined PMP and DP setting with bounded staleness.
• Global synchronization is reduced by aggregating updates per wave rather than per minibatch, lowering communication overhead.