[论文解读] On Index Policies for Restless Bandit Problems

Sudipto Guha, Kamesh Munagala+1 more|arXiv (Cornell University)|Nov 27, 2007

Advanced Bandit Algorithms Research参考文献 23被引用 9

一句话总结

本文提出了一种基于对偶的算法技术，为反馈MAB问题（一种仅在执行时才能观测到臂状态的 restless bandits 特例）提供了 2+ϵ-近似贪心策略。该方法首次实现了对非平凡 restless bandits 和 POMDP 实例的高效 O(1)-近似解，且可扩展至包含阻塞和切换成本等约束的变体。

ABSTRACT

The restless bandit problem is one of the most well-studied generalizations of the celebrated stochastic multi-armed bandit problem in decision theory. In its ultimate generality, the restless bandit problem is known to be PSPACE-Hard to approximate to any non-trivial factor, and little progress has been made despite its importance in modeling activity allocation under uncertainty. We consider a special case that we call Feedback MAB, where the reward obtained by playing each of n independent arms varies according to an underlying on/off Markov process whose exact state is only revealed when the arm is played. The goal is to design a policy for playing the arms in order to maximize the infinite horizon time average expected reward. This problem is also an instance of a Partially Observable Markov Decision Process (POMDP), and is widely studied in wireless scheduling and unmanned aerial vehicle (UAV) routing. Unlike the stochastic MAB problem, the Feedback MAB problem does not admit to greedy index-based optimal policies. We develop a novel and general duality-based algorithmic technique that yields a surprisingly simple and intuitive 2+epsilon-approximate greedy policy to this problem. We then define a general sub-class of restless bandit problems that we term Monotone bandits, for which our policy is a 2-approximation. Our technique is robust enough to handle generalizations of these problems to incorporate various side-constraints such as blocking plays and switching costs. This technique is also of independent interest for other restless bandit problems. By presenting the first (and efficient) O(1) approximations for non-trivial instances of restless bandits as well as of POMDPs, our work initiates the study of approximation algorithms in both these contexts.

研究动机与目标

• 为解决 restless bandits 问题的计算不可行性，这类问题近似求解属于 PSPACE-Hard。
• 为反馈MAB问题设计一种可计算、基于近似的方法，其中臂状态仅在被选择时才被揭示。
• 将该方法扩展至单调 bandits，并整合如阻塞和切换成本等附加约束。
• 开启对 restless bandits 和 POMDP 的近似算法研究，这些领域此前缺乏高效解法。

提出的方法

• 提出一种新颖的基于对偶的算法框架，用于推导 restless bandits 问题的近似策略。
• 应用对偶理论，为反馈MAB（一种部分可观测的马尔可夫决策过程）构建 2+ϵ-近似贪心策略。
• 识别出一类 restless bandits 的子类——单调 bandits，其策略可实现 2-近似。
• 将该框架推广以处理附加约束，包括阻塞操作和切换成本，同时保持近似保证。
• 利用对偶理论推导性能界，从而实现对近似最优策略的高效计算。
• 证明该技术具有鲁棒性，可广泛应用于反馈MAB以外的其他非平凡 restless bandits 和 POMDP 实例。

实验结果

研究问题

• RQ1 能否为反馈MAB问题设计一种可计算且高效的近似算法，该问题为 restless bandits 的非平凡实例？
• RQ2 尽管 restless bandits 问题属于 PSPACE-Hard，基于对偶的方法是否仍能实现常数因子近似？
• RQ3 该框架能否扩展以处理臂选择中的实际约束，如阻塞和切换成本？
• RQ4 是否存在一类自然的 restless bandits 子类——单调 bandits，使得 2-近似可实现？
• RQ5 该方法能否开启对 POMDP 和 restless bandits 问题中近似算法的更广泛研究？

主要发现

• 所提出的基于对偶的方法为反馈MAB问题提供了 2+ϵ-近似贪心策略，首次实现了对非平凡 restless bandits 实例的高效 O(1)-近似解。
• 对于单调 bandits 这一子类，策略实现了 2-近似，展示了强大的性能保证。
• 该框架成功整合了如阻塞操作和切换成本等附加约束，同时保持了近似界。
• 该技术具有通用性和鲁棒性，可广泛应用于反馈MAB以外的各类 restless bandits 问题。
• 该工作首次为非平凡 POMDP 实例建立了高效的 O(1)-近似解，为部分可观测决策问题中的近似算法研究开辟了新途径。
• 基于对偶的方法为在复杂不确定性序列决策问题中设计近似最优策略，提供了一种新颖、直观且计算可行的新方法。