[Paper Review] Thompson Sampling in Non-Episodic Restless Bandits

Young Hun Jung, Marc Abeille+1 more|arXiv (Cornell University)|Oct 12, 2019

Advanced Bandit Algorithms Research22 references4 citations

TL;DR

This paper proposes a variant of Thompson Sampling with dynamic episodes (TSDE) for non-episodic restless multi-armed bandits, where arm reward distributions evolve over time. By adapting TSDE to the restless bandit framework using a policy mapping, the authors establish a Bayesian regret bound of $\mathcal{O}(\sqrt{T}\log T)$, resolving an open question on sub-linear regret in the infinite-horizon setting.

ABSTRACT

Restless bandit problems assume time-varying reward distributions of the arms, which adds flexibility to the model but makes the analysis more challenging. We study learning algorithms over the unknown reward distributions and prove a sub-linear, $O(\sqrt{T}\log T)$, regret bound for a variant of Thompson sampling. Our analysis applies in the infinite time horizon setting, resolving the open question raised by Jung and Tewari (2019) whose analysis is limited to the episodic case. We adopt their policy mapping framework, which allows our algorithm to be efficient and simultaneously keeps the regret meaningful. Our algorithm adapts the TSDE algorithm of Ouyang et al. (2017) in a non-trivial manner to account for the special structure of restless bandits. We test our algorithm on a simulated dynamic channel access problem with several policy mappings, and the empirical regrets agree with the theoretical bound regardless of the choice of the policy mapping.

Motivation & Objective

• Address the open problem of achieving sub-linear regret in non-episodic restless bandits, where the system does not reset periodically.
• Develop an efficient learning algorithm that competes against a meaningful policy via a deterministic policy mapping $\mu$.
• Extend the TSDE algorithm to handle time-varying reward distributions in fully observable Markov decision processes.
• Ensure theoretical regret bounds hold under general conditions on the MDP's Bellman equation and mixing time.
• Demonstrate empirical robustness across different policy mappings, including Whittle index and myopic policies.

Proposed method

• Adapt the Thompson Sampling with Dynamic Episodes (TSDE) algorithm to the non-episodic restless bandit setting with time-varying arm rewards.
• Use a policy mapping $\mu$ that maps the unknown system parameter $\theta^\star$ to a deterministic stationary policy $\pi^\star = \mu(\theta^\star)$, enabling comparison against a meaningful baseline.
• Introduce two termination conditions per episode: one deterministic and one random, ensuring episode switching without periodic resets.
• At the start of each episode, sample a system parameter $\theta$ from the posterior distribution and execute the policy $\pi = \mu(\theta)$ for the duration of the episode.
• Leverage the policy mapping framework from Jung and Tewari (2019) to maintain computational efficiency while ensuring meaningful regret comparison.
• Establish theoretical regret bounds by decomposing the regret into components related to policy mismatch and parameter estimation, using concentration inequalities and mixing time analysis.

Experimental results

Research questions

• RQ1 Can Thompson Sampling achieve sub-linear regret in the non-episodic restless bandit setting, where the system does not reset periodically?
• RQ2 Does the TSDE algorithm maintain a meaningful regret bound when competing against a policy mapping $\mu(\theta^\star)$ in a time-varying environment?
• RQ3 How does the choice of policy mapping (e.g., Whittle index, myopic, fixed arm) affect the empirical performance of the algorithm?
• RQ4 Can the theoretical Bayesian regret bound of $\mathcal{O}(\sqrt{T}\log T)$ be empirically validated across different policy mappings and system parameters?
• RQ5 Does the algorithm learn the true system parameter $\theta^\star$ effectively, as indicated by posterior concentration?

Key findings

• The Bayesian regret of the proposed TSDE algorithm is bounded by $\mathcal{O}(\sqrt{T}\log T)$, resolving the open question on sub-linear regret in non-episodic restless bandits.
• Empirical results show sub-linear Bayesian regret across all tested policy mappings—best fixed arm, myopic, and Whittle index—confirming theoretical bounds.
• Log-log plots of regret versus time exhibit a slope of approximately 0.5, confirming the $\tilde{\mathcal{O}}(\sqrt{T})$ scaling predicted by theory.
• In the frequentist setting, the time-averaged cumulative rewards of TSDE converge to the benchmark values of $J_{\pi^\star}(\theta^\star)$, indicating effective learning.
• Posterior weights for the true parameters monotonically increase to one across all arms, demonstrating consistent learning of $\theta^\star$ under the Whittle index policy.
• The algorithm remains efficient and robust regardless of the choice of policy mapping, with no degradation in performance across different competitors.