[논문 리뷰] HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

Jay Park, Gyeongchan Yun외 6명|arXiv (Cornell University)|2020. 05. 28.

Advanced Neural Network Applications인용 수 41

한 줄 요약

HetPipe는 이질적 GPU에서 대형 DNN을 훈련하기 위해 파이프라인 모델 병렬성과 데이터 병렬성을 결합하고, 실험에서 최신 DP 대비 최대 49% 더 빠른 수렴을 달성한다.

ABSTRACT

Deep Neural Network (DNN) models have continuously been growing in size in order to improve the accuracy and quality of the models. Moreover, for training of large DNN models, the use of heterogeneous GPUs is inevitable due to the short release cycle of new GPU architectures. In this paper, we investigate how to enable training of large DNN models on a heterogeneous GPU cluster that possibly includes whimpy GPUs that, as a standalone, could not be used for training. We present a DNN training system, HetPipe (Heterogeneous Pipeline), that integrates pipelined model parallelism (PMP) with data parallelism (DP). In HetPipe, a group of multiple GPUs, called a virtual worker, processes minibatches in a pipelined manner, and multiple such virtual workers employ data parallelism for higher performance. We also propose a novel parameter synchronization model, which we refer to as Wave Synchronous Parallel (WSP) to accommodate both PMP and DP for virtual workers, and provide convergence proof of WSP. Our experimental results on a given heterogeneous setting show that with HetPipe, DNN models converge up to 49% faster compared to the state-of-the-art DP technique.

연구 동기 및 목표

• 혼합된(잠재적으로 형편없는) GPU를 가진 이질적 GPU 클러스터에서 대형 DNN 학습의 필요성을 제기한다.
• 가상 워커 내의 PMP와 가상 워커 간의 DP를 결합하여 효율적인 활용을 가능하게 한다.
• 이질적이고 파이프라인식 학습에 적합한 수렴 보장이 있는 동기화 모델을 제공한다.

제안 방법

• 이질성 하에서 데이터 병렬성을 가능하게 하기 위해 다수의 GPU로 구성된 가상 워커를 도입한다.
• 각 가상 워커 내에서 PMP를 위한 DNN 모델을 k 개의 파티션으로 분할하여 파이프라인을 형성한다.
• Wave Synchronous Parallel (WSP)을 파동별로 업데이트를 집계하는 수렴 보장을 갖는 동기화 모델로 제안한다.
• 글로벌 스태일니스가 한정된 상태에서 글로벌 가중치 동기화를 위해 파라미터 서버를 사용한다.
• WSP에 대한 수렴 증명을 제공한다.
• HetPipe를 구현하기 위해 TensorFlow를 수정하고 네 대 GPU 이질적 클러스터에서 평가한다.

실험 결과

연구 질문

• RQ1 Can large DNN models be trained efficiently on a heterogeneous GPU cluster by combining PMP and DP?
• RQ2 How should GPU resources be allocated and partitioned to maximize pipelined performance in HetPipe?
• RQ3 Does Wave Synchronous Parallel ensure convergence under heterogeneity and pipelined execution?
• RQ4 What performance gains are achieved over state-of-the-art DP approaches like Horovod with AllReduce?
• RQ5 How does HetPipe handle global and local staleness in a heterogeneous DP+PMP setting?

주요 결과

• HetPipe achieves faster convergence than Horovod-based DP by up to 49% for VGG-19 and 39% for ResNet-152 in their heterogeneous cluster setup.
• HetPipe enables training of large models that could be too big for a single whimpy GPU by forming virtual workers.
• PMP within virtual workers and DP across virtual workers improve utilization of heterogeneous GPUs.
• WSP provides a convergence guarantee for the combined PMP and DP setting with bounded staleness.
• Global synchronization is reduced by aggregating updates per wave rather than per minibatch, lowering communication overhead.