[Paper Review] Deadline Scheduling as Restless Bandits

Zhe Yu, Yunjian Xu+1 more|arXiv (Cornell University)|Oct 3, 2016

Advanced Wireless Network Optimization43 references3 citations

TL;DR

This paper formulates stochastic deadline scheduling as a restless multi-armed bandit (RMAB) problem and establishes its indexability. It derives a closed-form Whittle's index for constant processing costs and proves that the gap-to-optimality of the Whittle index policy vanishes asymptotically as the job arrival rate and number of processors grow large, demonstrating asymptotic optimality in heavy traffic.

ABSTRACT

The problem of stochastic deadline scheduling is considered. A constrained Markov decision process model is introduced in which jobs arrive randomly at a service center with stochastic job sizes, rewards, and completion deadlines. The service provider faces random processing costs, convex non-completion penalties, and a capacity constraint that limits the simultaneous processing of jobs. Formulated as a restless multi-armed bandit problem, the stochastic deadline scheduling problem is shown to be indexable. A closed-form expression of the Whittle's index is obtained for the case when the processing costs are constant. An upper bound on the gap-to-optimality for the Whittle's index policy is obtained, and it is shown that the bound converges to zero as the job arrival rate and the number of available processors increase simultaneously to infinity.

Motivation & Objective

• To model stochastic deadline scheduling as a constrained Markov decision process with random job arrivals, sizes, deadlines, and processing costs.
• To reformulate the problem as a restless multi-armed bandit (RMAB) to enable index-based scheduling policies.
• To establish indexability of the RMAB formulation and derive a closed-form Whittle's index under constant processing costs.
• To bound the performance gap of the Whittle index policy and analyze its asymptotic optimality as system load increases.
• To demonstrate that the gap-to-optimality converges to zero in the light traffic regime when both arrival rate and number of processors grow.

Proposed method

• Formulates the deadline scheduling problem as a constrained MDP with discounted reward maximization under a simultaneous processing capacity constraint.
• Reframes the MDP as a restless multi-armed bandit (RMAB) problem with simultaneous plays, enabling index-based control.
• Proves indexability of the RMAB formulation by establishing monotonicity and concavity of the value function in the number of jobs.
• Derives a closed-form expression for Whittle's index under constant processing costs, generalizing prior results.
• Bounded the gap-to-optimality using conditional value at risk (CVaR) of the arrival process, enabling performance analysis.
• Uses induction and dynamic programming recursion to prove monotonicity and concavity of the Whittle index and value function.

Experimental results

Research questions

• RQ1 Is the stochastic deadline scheduling problem indexable when modeled as a restless multi-armed bandit?
• RQ2 Can a closed-form Whittle's index be derived for the case of constant processing costs?
• RQ3 What is the performance gap between the Whittle index policy and the optimal policy, and how does it scale with system load?
• RQ4 Does the Whittle index policy become asymptotically optimal as the number of processors and job arrival rate increase?
• RQ5 How does the conditional value at risk (CVaR) of the arrival process relate to the performance loss of the index policy?

Key findings

• The RMAB formulation of the deadline scheduling problem is indexable, enabling the use of Whittle's index policy.
• A closed-form expression for Whittle's index is derived when processing costs are constant, simplifying online computation.
• The gap-to-optimality of the Whittle index policy is bounded by the CVaR of the number of job arrivals per unit time.
• The performance gap converges to zero in the light traffic regime as both the job arrival rate and number of processors increase to infinity.
• The Whittle index policy is asymptotically optimal under heavy traffic conditions, particularly when the system load grows proportionally.
• The proof relies on induction to establish monotonicity of the Whittle index and concavity of the value function in the number of jobs.