[Paper Review] Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

Myeongjae Jeon, Shivaram Venkataraman+4 more|arXiv (Cornell University)|Jan 17, 2019

Cloud Computing and Resource ManagementComputer Science118 citations

TL;DR

The paper analyzes a large Microsoft multi-tenant GPU cluster to understand locality, scheduling, and failure effects on DNN training, and offers design guidelines for next-gen schedulers.

ABSTRACT

With widespread advances in machine learning, a number of large enterprises are beginning to incorporate machine learning models across a number of products. These models are typically trained on shared, multi-tenant GPU clusters. Similar to existing cluster computing workloads, scheduling frameworks aim to provide features like high efficiency, resource isolation, fair sharing across users, etc. However Deep Neural Network (DNN) based workloads, predominantly trained on GPUs, differ in two significant ways from traditional big data analytics workloads. First, from a cluster utilization perspective, GPUs represent a monolithic resource that cannot be shared at a fine granularity across users. Second, from a workload perspective, deep learning frameworks require gang scheduling reducing the flexibility of scheduling and making the jobs themselves inelastic to failures at runtime. In this paper we present a detailed workload characterization of a two-month long trace from a multi-tenant GPU cluster in a large enterprise. By correlating scheduler logs with logs from individual jobs, we study three distinct issues that affect cluster utilization for DNN training workloads on multi-tenant clusters: (1) the effect of gang scheduling and locality constraints on queuing, (2) the effect of locality on GPU utilization, and (3) failures during training. Based on our experience running a large-scale operation, we provide design guidelines pertaining to next-generation cluster schedulers for DNN training workloads.

Motivation & Objective

• Characterize how gang scheduling and locality constraints affect queueing and utilization in a large multi-tenant GPU cluster used for DNN training.
• Assess how GPU locality, server-level interference, and co-location impact GPU utilization and training efficiency.
• Identify common failure modes in DNN training workloads and their impact on cluster utilization and retry policies.
• Provide design guidelines for next-generation cluster schedulers to improve locality, isolation, and early failure detection in DNN workloads.

Proposed method

• Analyze a two-month trace from a Microsoft multi-tenant GPU cluster (Philly) covering ~100,000 jobs and 14 virtual clusters.
• Correlate scheduler logs (YARN) with per-job logs and Ganglia utilization data to study locality, queueing, and failures.
• Characterize queueing delays into fair-share and fragmentation components and their dependency on GPU counts.
• Evaluate GPU and host resource utilization under different placement scenarios (same-server, diff-server, intra-/inter-server).
• Compare Philly with other schedulers and provide practical design guidelines for locality-aware scheduling in DNN workloads.

Experimental results

Research questions

• RQ1 How do locality constraints and gang scheduling affect queueing delays for DNN training jobs?
• RQ2 How does locality-aware scheduling influence GPU utilization and training performance across distributed multi-GPU jobs?
• RQ3 What are the primary causes of job failures in a large multi-tenant DNN training cluster and how do they impact utilization?
• RQ4 What scheduler design choices can mitigate fragmentation, interference, and failures to improve utilization and performance?

Key findings

• Queueing delays are influenced by locality, with relaxing locality reducing delays, especially for larger GPU jobs (>4 GPUs).
• Average GPU hardware utilization across in-use GPUs is only about 52%, with underutilization rising for larger jobs due to synchronization and interference.
• Fragmentation delays dominate waiting times for many jobs, especially 5–8 GPU and larger configurations; fair-share delays occur when quotas are exhausted.
• Around 30% of jobs end unsuccessfully or are killed, yet they account for a substantial portion of GPU time, highlighting inefficiencies from failures.
• Distributed training across multiple servers reduces GPU utilization due to RDMA/PCIe contention and inter-server communication overhead; colocated jobs further degrade utilization.
• Most passed jobs require nearly all epochs to reach the best loss, suggesting potential early-termination opportunities to save GPU time.