[论文解读] Markovian restless bandits and index policies: A review

Niño-Mora, José|arXiv (Cornell University)|Jan 19, 2026

Advanced Bandit Algorithms Research被引用 0

一句话总结

本文综述了不安定多臂赌博问题（RMABP）与Whittle指数策略，概述理论、算法与应用，同时讨论指数性、计算与渐近结果。

ABSTRACT

The restless multi-armed bandit problem is a paradigmatic modeling framework for optimal dynamic priority allocation in stochastic models of wide-ranging applications that has been widely investigated and applied since its inception in a seminal paper by Whittle in the late 1980s. The problem has generated a vast and fast-growing literature from which a significant sample is thematically organized and reviewed in this paper. While the main focus is on priority-index policies due to their intuitive appeal, tractability, asymptotic optimality properties, and often strong empirical performance, other lines of work are also reviewed. Theoretical and algorithmic developments are discussed, along with diverse applications. The main goals are to highlight the remarkable breadth of work that has been carried out on the topic and to stimulate further research in the field.

研究动机与目标

• Motivate and map the RMABP framework and its breadth of applications.
• Highlight index policies, especially Whittle’s, for tractability and practical performance.
• Review complexity, relaxations, and indexability conditions underlying RMABP solutions.
• Summarize computation methods for Whittle indices and related policies.
• Survey extensions to learning, online adaptation, and multi-action bandits.

提出的方法

• Describe the RMABP setup with N projects, states, actions, rewards, and the average/discount criteria.
• Explain Whittle’s Lagrangian relaxation leading to projectwise subproblems and the index notion.
• Discuss indexability as a condition for the Whittle index to exist and be computable.
• Summarize complexity results, LP relaxations, and approximate/heuristic policies.
• Outline computation algorithms for Whittle indices, including PCL-based and pivoting methods.
• Review extensions to multi-action bandits, fluid relaxations, and learning approaches.

实验结果

研究问题

• RQ1 What is the RMABP and how does Whittle’s index policy arise from a Lagrangian relaxation?
• RQ2 Under what conditions is indexability satisfied and how can Whittle indices be computed?
• RQ3 What are the complexity results and approximation guarantees for RMABP solutions?
• RQ4 How do learning, online learning, and reinforcement learning approaches apply to RMABP?
• RQ5 What are the main applications and extensions to multi-action and POMDP RMABPs?

主要发现

• RMABP substantial expands modeling power beyond classical MABP by allowing restless state transitions, at the cost of computational intractability.
• Whittle’s index policy is a practical heuristic that often performs well and has asymptotic optimality under certain conditions; indexability is crucial for its validity.
• There exist constant-factor approximation algorithms and LP-based relaxations that yield performance guarantees for RMABP variants.
• Index computation methods include closed-form forms, adaptive-greedy, and O(n^3) pivoting algorithms, with special cases enabling faster O(n) or simpler checks for indexability.
• Reinforcement learning, Q-learning, and online learning frameworks are actively applied to RMABP, broadening practical applicability.
• Multi-action RMABPs and fluid/Lagrangian relaxations extend the framework and broaden the scope of index-based policies.