[Paper Review] Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption

Yige Hong, Qiaomin Xie+2 more|arXiv (Cornell University)|May 31, 2023

Advanced Bandit Algorithms ResearchDecision Sciences3 citations

TL;DR

This paper introduces the Follow-the-Virtual-Advice (FTVA) framework, a simulation-based method that converts any single-armed policy into an N-armed policy for restless bandits under the average reward criterion. It achieves an $O(1/ ext{sqrt}{N})$ optimality gap in both discrete- and continuous-time settings without requiring the Uniform Global Attractor Property (UGAP), a key assumption in prior work, thus providing the first asymptotic optimality result under weaker conditions in the continuous-time case.

ABSTRACT

We study the infinite-horizon restless bandit problem with the average reward criterion, in both discrete-time and continuous-time settings. A fundamental goal is to efficiently compute policies that achieve a diminishing optimality gap as the number of arms, $N$, grows large. Existing results on asymptotic optimality all rely on the uniform global attractor property (UGAP), a complex and challenging-to-verify assumption. In this paper, we propose a general, simulation-based framework, Follow-the-Virtual-Advice, that converts any single-armed policy into a policy for the original $N$-armed problem. This is done by simulating the single-armed policy on each arm and carefully steering the real state towards the simulated state. Our framework can be instantiated to produce a policy with an $O(1/\sqrt{N})$ optimality gap. In the discrete-time setting, our result holds under a simpler synchronization assumption, which covers some problem instances that violate UGAP. More notably, in the continuous-time setting, we do not require \emph{any} additional assumptions beyond the standard unichain condition. In both settings, our work is the first asymptotic optimality result that does not require UGAP.

Motivation & Objective

• To address the fundamental open question of whether asymptotic optimality is achievable without the Uniform Global Attractor Property (UGAP) in infinite-horizon restless bandits under the average reward criterion.
• To establish a non-trivial convergence rate for the optimality gap in the absence of UGAP and non-singularity assumptions.
• To develop a general, efficient, and scalable policy conversion framework that reduces the N-armed problem to single-armed policy design.
• To extend the framework to heterogeneous arms and demonstrate its implementability with linear-in-N computational cost and distributed subroutines.

Proposed method

• Proposes the Follow-the-Virtual-Advice (FTVA) framework, which simulates a single-armed policy on each arm in parallel and steers the real system state toward the simulated state to minimize performance loss.
• Uses a synchronization mechanism where each arm's virtual and real processes are aligned via a synchronization time $\tau^{\text{sync}}_k(s,a,\widehat{s},\widehat{a})$, ensuring state alignment under the Synchronization Assumption (SA).
• Applies Little’s Law to bound the average disagreement period length using expected synchronization times, and uses Cauchy-Schwarz to bound the expected number of mismatched arms.
• Derives an optimality gap upper bound of $\frac{r_{\max}\overline{\tau}^{\text{sync}}_{\text{max}}}{\sqrt{N}}$ under the Synchronization Assumption in discrete time and unichain condition in continuous time.
• Extends the framework to heterogeneous arms by applying type-specific optimal single-armed policies $\bar{\pi}^*_k$ and aggregating results via weighted averaging.
• Introduces a continuous-time variant, FTVA-CT, which achieves the same $O(1/\sqrt{N})$ gap without any assumption beyond unichain, marking the first such result in continuous time without UGAP.

Figure 1 : An discrete-time RB problem that satisfies SA but not UGAP.

Experimental results

Research questions

• RQ1 Can asymptotic optimality be achieved in restless bandits with average reward without relying on the Uniform Global Attractor Property (UGAP)?
• RQ2 Is it possible to establish a non-trivial convergence rate for the optimality gap when UGAP and non-singularity conditions are relaxed?
• RQ3 Can a general, simulation-based framework convert any single-armed policy into an asymptotically optimal N-armed policy under weaker assumptions?
• RQ4 How does the performance gap scale with the number of arms $N$ in the absence of UGAP, particularly in continuous-time settings?

Key findings

• The FTVA framework achieves an $O(1/\sqrt{N})$ optimality gap in discrete-time restless bandits under the Synchronization Assumption (SA), which is strictly weaker than UGAP and covers instances where UGAP fails.
• In continuous-time settings, the FTVA-CT variant achieves the same $O(1/\sqrt{N})$ optimality gap without any assumption beyond the standard unichain condition, marking the first such result without UGAP.
• The optimality gap is bounded by $\frac{r_{\max}\overline{\tau}^{\text{sync}}_{\text{max}}}{\sqrt{N}}$, where $\overline{\tau}^{\text{sync}}_{\text{max}}$ is the maximum expected synchronization time across all arm types and initial conditions.
• The framework enables efficient computation: the policy can be implemented with $O(N)$ computational cost and supports distributed execution across arms.
• The method is extendable to heterogeneous arms by applying type-specific optimal single-armed policies and aggregating via weighted averaging.
• The work resolves two long-standing open questions: asymptotic optimality without UGAP, and non-trivial convergence rates without non-singularity assumptions.

Figure 2 : Time evolution of the fraction of arms in each state under LP-Priority (upper), or after switching to $\textup{{FTVA}}({\bar{\pi}}^{*})$ (lower) since time slot $250$ . The x-axis represents the time slot, which ranges from $250$ to $289$ ; the y-axis represents the states; the color repr