[Paper Review] Markovian restless bandits and index policies: A review

Niño-Mora, José|arXiv (Cornell University)|Jan 19, 2026

Advanced Bandit Algorithms Research0 citations

TL;DR

This paper surveys the restless multi-armed bandit problem (RMABP) and the Whittle index policy, summarizing theory, algorithms, and applications while discussing indexability, computations, and asymptotic results.

ABSTRACT

The restless multi-armed bandit problem is a paradigmatic modeling framework for optimal dynamic priority allocation in stochastic models of wide-ranging applications that has been widely investigated and applied since its inception in a seminal paper by Whittle in the late 1980s. The problem has generated a vast and fast-growing literature from which a significant sample is thematically organized and reviewed in this paper. While the main focus is on priority-index policies due to their intuitive appeal, tractability, asymptotic optimality properties, and often strong empirical performance, other lines of work are also reviewed. Theoretical and algorithmic developments are discussed, along with diverse applications. The main goals are to highlight the remarkable breadth of work that has been carried out on the topic and to stimulate further research in the field.

Motivation & Objective

• Motivate and map the RMABP framework and its breadth of applications.
• Highlight index policies, especially Whittle’s, for tractability and practical performance.
• Review complexity, relaxations, and indexability conditions underlying RMABP solutions.
• Summarize computation methods for Whittle indices and related policies.
• Survey extensions to learning, online adaptation, and multi-action bandits.

Proposed method

• Describe the RMABP setup with N projects, states, actions, rewards, and the average/discount criteria.
• Explain Whittle’s Lagrangian relaxation leading to projectwise subproblems and the index notion.
• Discuss indexability as a condition for the Whittle index to exist and be computable.
• Summarize complexity results, LP relaxations, and approximate/heuristic policies.
• Outline computation algorithms for Whittle indices, including PCL-based and pivoting methods.
• Review extensions to multi-action bandits, fluid relaxations, and learning approaches.

Experimental results

Research questions

• RQ1 What is the RMABP and how does Whittle’s index policy arise from a Lagrangian relaxation?
• RQ2 Under what conditions is indexability satisfied and how can Whittle indices be computed?
• RQ3 What are the complexity results and approximation guarantees for RMABP solutions?
• RQ4 How do learning, online learning, and reinforcement learning approaches apply to RMABP?
• RQ5 What are the main applications and extensions to multi-action and POMDP RMABPs?

Key findings

• RMABP substantially expands modeling power beyond classical MABP by allowing restless state transitions, at the cost of computational intractability.
• Whittle’s index policy is a practical heuristic that often performs well and has asymptotic optimality under certain conditions; indexability is crucial for its validity.
• There exist constant-factor approximation algorithms and LP-based relaxations that yield performance guarantees for RMABP variants.
• Index computation methods include closed-form forms, adaptive-greedy, and O(n^3) pivoting algorithms, with special cases enabling faster O(n) or simpler checks for indexability.
• Reinforcement learning, Q-learning, and online learning frameworks are actively applied to RMABP, broadening practical applicability.
• Multi-action RMABPs and fluid/Lagrangian relaxations extend the framework and broaden the scope of index-based policies.