[Paper Review] Themis: Fair and Efficient GPU Cluster Scheduling

Kshiteej Mahajan, Arjun Balasubramanian+5 more|arXiv (Cornell University)|Jul 2, 2019

Stochastic Gradient Optimization Techniques36 references22 citations

TL;DR

Themis proposes a fair and efficient GPU cluster scheduler for machine learning workloads by introducing finish-time fairness—a long-term fairness metric that ensures each ML application completes in a time comparable to running on a private cluster. It uses a two-level auction-based scheduling architecture where apps bid on GPU resources with placement preferences, enabling both short-term efficiency and long-term fairness through a partial allocation auction that incentivizes truthful bidding.

ABSTRACT

Modern distributed machine learning (ML) training workloads benefit significantly from leveraging GPUs. However, significant contention ensues when multiple such workloads are run atop a shared cluster of GPUs. A key question is how to fairly apportion GPUs across workloads. We find that established cluster scheduling disciplines are a poor fit because of ML workloads' unique attributes: ML jobs have long-running tasks that need to be gang-scheduled, and their performance is sensitive to tasks' relative placement. We propose Themis, a new scheduling framework for ML training workloads. It's GPU allocation policy enforces that ML workloads complete in a finish-time fair manner, a new notion we introduce. To capture placement sensitivity and ensure efficiency, Themis uses a two-level scheduling architecture where ML workloads bid on available resources that are offered in an auction run by a central arbiter. Our auction design allocates GPUs to winning bids by trading off efficiency for fairness in the short term but ensuring finish-time fairness in the long term. Our evaluation on a production trace shows that Themis can improve fairness by more than 2.25X and is ~5% to 250% more cluster efficient in comparison to state-of-the-art schedulers.

Motivation & Objective

• Address the inadequacy of existing cluster schedulers (e.g., DRF, Quincy) in handling long-running, gang-scheduled ML workloads with placement sensitivity.
• Introduce finish-time fairness as a new long-term fairness metric that ensures each ML application performs as well as if it had exclusive access to 1/N of the cluster.
• Design a scheduling framework that balances short-term efficiency with long-term fairness while respecting placement preferences of ML jobs.
• Ensure Pareto efficiency and envy-freeness in resource allocation through a truthful auction mechanism.
• Enable practical deployment via a two-level scheduling architecture with a centralized arbiter and application-level bidding rounds.

Proposed method

• Introduce a round-by-round auction mechanism triggered at lease expiration, where apps bid on available GPU resources based on their projected finish-time fairness.
• Use a partial allocation auction to ensure truthfulness, Pareto efficiency, and envy-freeness by design, with hidden payments that discourage strategic misreporting.
• Allow apps to express placement preferences (e.g., dense or staggered placement) via bid values that reflect expected performance under different GPU configurations.
• Implement a two-level scheduling architecture: a centralized inter-app scheduler (arbiter) and a per-app intra-app scheduler that manages lease-based task execution.
• Filter out apps with the worst finish-time fairness metrics (based on fairness knob f) to prioritize those most in need, improving overall fairness.
• Use lease-based scheduling (10-minute default) to enable checkpointing and resource reallocation while minimizing overhead.

Experimental results

Research questions

• RQ1 How can fairness be meaningfully defined for long-running, gang-scheduled ML workloads with placement sensitivity?
• RQ2 Can a scheduling system achieve both short-term efficiency and long-term fairness without sacrificing placement-aware performance?
• RQ3 How can truthful bidding be incentivized in a dynamic, multi-round GPU auction for ML workloads?
• RQ4 What is the impact of lease duration and fairness knob tuning on fairness and cluster utilization?
• RQ5 To what extent does the proposed auction mechanism outperform state-of-the-art schedulers in fairness and efficiency?

Key findings

• Themis improves fairness by more than 2.25× compared to state-of-the-art schedulers on a production trace.
• Themis achieves 5% to 250% higher cluster efficiency than existing schedulers, depending on workload characteristics.
• The partial allocation auction mechanism successfully incentivizes truthful bidding, as lying strategies only benefit the liar up to a point (X ≤ 34%), after which the liar loses significant resources.
• Setting the fairness knob f = 0.8 and lease time = 10 minutes provides the best trade-off between fairness and efficiency, avoiding degradation at f = 1.0 due to reduced bidding competition.
• Smaller lease times improve fairness by reducing queueing delays, while higher lease values improve efficiency by reducing checkpointing frequency.
• The system maintains Pareto efficiency and envy-freeness by design, ensuring no app can improve its outcome without harming another.