[Paper Review] Themis: Fair and Efficient GPU Cluster Scheduling for Machine Learning Workloads

Kshiteej Mahajan, Arjun Singhvi+7 more|arXiv (Cornell University)|Jul 2, 2019

Stochastic Gradient Optimization TechniquesComputer Science8 citations

TL;DR

Themis is a novel GPU cluster scheduling framework for machine learning workloads that ensures finish-time fairness through a two-level auction-based allocation system. By modeling GPU allocation as a dynamic auction where bids are prioritized based on fairness and efficiency trade-offs, Themis achieves higher cluster utilization and improved fairness compared to state-of-the-art schedulers.

ABSTRACT

Modern distributed machine learning (ML) training workloads benefit significantly from leveraging GPUs. However, significant contention ensues when multiple such workloads are run atop a shared cluster of GPUs. A key question is how to fairly apportion GPUs across workloads while ensuring overall cluster efficiency. We find that established cluster scheduling disciplines that provide instantaneous fair share of resources are a poor fit because of ML workloads' unique attributes. ML jobs are typically long running, have coarse grained tasks that need to be gang-scheduled, and their performance is sensitive to tasks' relative placement. These properties cannot be captured by existing fair sharing schemes. We propose Themis, a new scheduling framework for ML training workloads. It's GPU allocation policy enforces that ML workloads complete in a finish-time fair manner, a new notion we introduce. To capture placement sensitivity and ensure efficiency, Themis uses a two-level scheduling architecture where ML workloads bid on available resources that are offered in an auction run by a central arbiter. Our auction design allocates GPUs to winning bids by trading off efficiency for fairness in the short term but compensating for finish-time fairness in the long term. Our evaluation on a number of machine learning models shows that Themis can ensure greater fairness while providing more efficient allocations compared to state-of-the-art schedulers.

Motivation & Objective

• To address the limitations of existing fair-share scheduling in GPU clusters for long-running, gang-scheduled ML workloads.
• To design a scheduling framework that accounts for placement sensitivity and task co-location requirements in ML training.
• To achieve finish-time fairness—where workloads complete in proportion to their resource entitlement—while maintaining high cluster efficiency.
• To develop a two-level scheduling architecture that separates resource bidding from centralized arbitration to scale across large clusters.
• To evaluate the framework against state-of-the-art schedulers using real ML workloads and quantify fairness and efficiency improvements.

Proposed method

• Themis introduces a new fairness metric called finish-time fairness, which ensures workloads complete in proportion to their resource shares.
• It employs a two-level scheduling architecture: local schedulers on each node collect bids from ML jobs, and a central arbiter runs an auction to allocate GPUs.
• The auction mechanism prioritizes bids based on a composite score balancing urgency, fairness, and efficiency, with short-term efficiency gains offset by long-term fairness compensation.
• GPU allocation decisions are made using a dynamic bidding system where jobs submit bids reflecting their resource needs and priority.
• The system enforces gang scheduling by ensuring all required GPUs for a job are allocated together, preserving placement sensitivity.
• A feedback loop adjusts future bids based on past performance and fairness metrics to maintain long-term balance.

Experimental results

Research questions

• RQ1 Can a scheduling framework achieve finish-time fairness for ML workloads in shared GPU clusters without sacrificing cluster efficiency?
• RQ2 How do placement sensitivity and gang scheduling constraints affect the design of fair and efficient GPU schedulers?
• RQ3 To what extent can auction-based mechanisms balance short-term efficiency with long-term fairness in ML cluster scheduling?
• RQ4 How does the proposed two-level architecture scale and maintain responsiveness in large-scale GPU clusters?
• RQ5 What performance improvements does Themis offer over existing schedulers in terms of fairness and resource utilization?

Key findings

• Themis achieves significantly higher finish-time fairness compared to state-of-the-art schedulers, ensuring workloads complete in proportion to their resource entitlements.
• The framework improves cluster efficiency by reducing idle GPU time and better matching resource supply with demand through dynamic bidding.
• By enforcing gang scheduling, Themis maintains high performance for ML workloads that are sensitive to task placement and co-location.
• The auction-based allocation mechanism enables short-term efficiency gains while compensating for fairness deviations over time, ensuring long-term balance.
• Evaluation on diverse ML models shows that Themis outperforms existing schedulers in both fairness and utilization metrics across multiple benchmark scenarios.