[Paper Review] Problem Dependent Reinforcement Learning Bounds Which Can Identify Bandit Structure in MDPs

Andrea Zanette, Emma Brunskill|arXiv (Cornell University)|Nov 3, 2019

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes a minor variant of the UCRL2 reinforcement learning algorithm that automatically achieves near-optimal regret bounds of $\tilde{O}(\sqrt{SAT})$ in tabular contextual bandit problems, even when the agent is unaware of the bandit structure. The method leverages problem-dependent analysis to inherit the best performance of simpler frameworks without user intervention or algorithm modification.

ABSTRACT

In order to make good decision under uncertainty an agent must learn from observations. To do so, two of the most common frameworks are Contextual Bandits and Markov Decision Processes (MDPs). In this paper, we study whether there exist algorithms for the more general framework (MDP) which automatically provide the best performance bounds for the specific problem at hand without user intervention and without modifying the algorithm. In particular, it is found that a very minor variant of a recently proposed reinforcement learning algorithm for MDPs already matches the best possible regret bound $ ilde O (\sqrt{SAT})$ in the dominant term if deployed on a tabular Contextual Bandit problem despite the agent being agnostic to such setting.

Motivation & Objective

• To develop reinforcement learning algorithms that automatically inherit the best performance bounds of simpler decision frameworks (e.g., contextual bandits) when the true problem structure is simpler than assumed.
• To address the challenge of framework mismatch in RL, where choosing between MDPs, bandits, or POMDPs is often unclear and impacts learning efficiency.
• To provide problem-dependent regret bounds that adapt to the underlying structure (e.g., bandit vs. MDP) without requiring users to specify the framework in advance.
• To close the gap between theoretical bounds for MDPs and contextual bandits, particularly in the dominant $\sqrt{SAT}$ term.

Proposed method

• A small modification to the UCRL2 algorithm is introduced to improve its regret analysis in settings where the true problem is a contextual bandit.
• The method uses a problem-dependent analysis that conditions on hitting times and visit counts to derive tighter bounds.
• It applies Markov and Hoeffding inequalities to bound the probability of excessive state visits, leveraging geometric random variable approximations.
• The analysis leverages the maximum mean hitting time $T_{\text{hit}}$ and the number of visits to state-action pairs to control regret growth.
• It proves that the expected number of visits to a state-action pair is bounded by $T_{\text{hit}} \cdot \max\{1, N_k(s_{k_2}, \pi(s_{k_2}))\}$, enabling tighter regret control.
• The algorithm's performance is shown to scale as $\tilde{O}(\sqrt{SAT})$ in the dominant term when deployed on a tabular contextual bandit, matching the best-known bounds for bandit-specific algorithms.

Experimental results

Research questions

• RQ1 Can an MDP-based RL algorithm automatically achieve the best possible regret bound of $\tilde{O}(\sqrt{SAT})$ when the true environment is a tabular contextual bandit?
• RQ2 Does a problem-dependent analysis of UCRL2 allow it to adapt to simpler structures like bandits without explicit modification or user knowledge of the framework?
• RQ3 Can the regret bound of an MDP algorithm be made independent of the horizon $H$ when the environment is actually a bandit, thus matching bandit-specific bounds?
• RQ4 What role does the maximum mean hitting time $T_{\text{hit}}$ play in controlling regret in bandit-like environments under MDP algorithms?
• RQ5 How can tighter regret bounds be derived for MDP algorithms by exploiting structural properties such as state visitation patterns and policy stability?

Key findings

• A small variant of UCRL2 achieves a regret bound of $\tilde{O}(\sqrt{SAT})$ in tabular contextual bandit problems, matching the best-known bound for bandit-specific algorithms.
• The algorithm automatically inherits the performance of contextual bandits without requiring the user to identify or specify the bandit structure in advance.
• The regret bound is derived using problem-dependent analysis that conditions on hitting times and visit counts, leading to tighter control over state visitation.
• The expected number of visits to a state-action pair is bounded by $T_{\text{hit}} \cdot \max\{1, N_k(s_{k_2}, \pi(s_{k_2}))\}$, which is critical for deriving the final bound.
• The method uses geometric random variable approximations and Hoeffding's inequality to bound tail probabilities of visit counts, enabling high-probability regret control.
• The analysis shows that the algorithm's performance degrades gracefully when the environment is simpler than assumed, achieving optimal scaling in the dominant term for bandit settings.