[Paper Review] Bounded regret in stochastic multi-armed bandits

Sébastien Bubeck, Vianney Perchet+1 more|arXiv (Cornell University)|Feb 6, 2013

Advanced Bandit Algorithms Research11 references21 citations

TL;DR

This paper introduces a randomized policy for stochastic multi-armed bandits that achieves uniformly bounded regret when both the optimal mean reward $\mu^{(\star)}$ and a lower bound $\Delta$ on the smallest positive gap are known. The policy uses sequential likelihood ratio testing and achieves regret bounded by $\Delta + 16/\Delta$ in the two-armed case, with generalization to $K$ arms showing regret scaling as $\sum_{i:\Delta_i>0} \left\{ \Delta_i + \frac{32}{\Delta_i} \log\left(\frac{5}{\varepsilon}\right) \right\}$ when $\varepsilon$ is a lower bound on $\Delta$. The key contribution is proving that bounded regret is impossible with only $\Delta$ or only $\mu^{(\star)}$, establishing minimax optimality of the proposed approach.

ABSTRACT

We study the stochastic multi-armed bandit problem when one knows the value $μ^{(\star)}$ of an optimal arm, as a well as a positive lower bound on the smallest positive gap $Δ$. We propose a new randomized policy that attains a regret {\em uniformly bounded over time} in this setting. We also prove several lower bounds, which show in particular that bounded regret is not possible if one only knows $Δ$, and bounded regret of order $1/Δ$ is not possible if one only knows $μ^{(\star)}$

Motivation & Objective

• To investigate whether bounded regret is achievable in stochastic multi-armed bandits when partial prior knowledge—specifically $\mu^{(\star)}$ and a lower bound on $\Delta$—is available.
• To design a randomized policy that achieves uniformly bounded regret under these conditions, particularly in the two-armed case.
• To establish tight lower bounds showing that bounded regret is impossible if only $\Delta$ or only $\mu^{(\star)}$ is known, even in the two-armed setting.
• To generalize the policy to $K$-armed bandits and analyze regret dependence on $\varepsilon$, a lower bound on $\Delta$, showing logarithmic scaling in $1/\varepsilon$.

Proposed method

• Proposes a randomized policy based on sequential likelihood ratio testing for the two-armed bandit problem with known $\mu^{(\star)}$ and $\Delta$, ensuring bounded regret.
• Uses a randomized exploration strategy that balances confidence in arm selection with the known optimal mean and gap.
• Derives an upper bound on regret of $\Delta + 16/\Delta$ for the two-armed case, uniformly over time horizon $n$, using concentration inequalities and KL divergence bounds.
• Generalizes the policy to $K$ arms by applying the same principle with a modified exploration schedule, leading to regret $\sum_{i:\Delta_i>0} \left\{ \Delta_i + \frac{32}{\Delta_i} \log\left(\frac{5}{\varepsilon}\right) \right\}$ when $\varepsilon$ is a lower bound on $\Delta_i$.
• Introduces an improved policy where $\log(1/\varepsilon)$ is replaced by $\log(\Delta_i/\varepsilon)\log\log(1/\varepsilon)$, reducing logarithmic dependence when $\Delta_i$ and $\varepsilon$ are of the same order.
• Employs information-theoretic tools, including Kullback-Leibler divergence and Hellinger affinity, to derive non-asymptotic lower bounds on regret.

Experimental results

Research questions

• RQ1 Can uniformly bounded regret be achieved in stochastic multi-armed bandits when both the optimal mean $\mu^{(\star)}$ and a lower bound on the smallest positive gap $\Delta$ are known?
• RQ2 Is bounded regret possible if only $\Delta$ is known, or only $\mu^{(\star)}$ is known, in the two-armed bandit setting?
• RQ3 What is the optimal dependence of regret on $\varepsilon$, a lower bound on $\Delta$, in the $K$-armed case when $\mu^{(\star)}$ is known?
• RQ4 Can the logarithmic dependence on $1/\varepsilon$ in the regret bound be improved when $\varepsilon$ is close to $\Delta_i$?
• RQ5 What is the minimax lower bound on regret when only $\Delta$ or only $\mu^{(\star)}$ is known, and how does this compare to the upper bound of the proposed policy?

Key findings

• The proposed policy achieves uniformly bounded regret of $\Delta + 16/\Delta$ in the two-armed bandit problem when both $\mu^{(\star)}$ and $\Delta$ are known.
• Any policy that knows only $\Delta$ must incur a rescaled regret $\Delta R_n$ that grows at least logarithmically in $n$, proving bounded regret is impossible with only $\Delta$.
• Any policy that knows only $\mu^{(\star)}$ must incur a rescaled regret $\Delta R_n$ of at least order $\log n$, showing bounded regret is impossible with only $\mu^{(\star)}$.
• For the $K$-armed bandit problem with known $\mu^{(\star)}$ and a lower bound $\varepsilon$ on $\Delta$, the policy achieves regret $\sum_{i:\Delta_i>0} \left\{ \Delta_i + \frac{32}{\Delta_i} \log\left(\frac{5}{\varepsilon}\right) \right\}$, which is tight up to constants.
• An improved policy reduces the $\log(1/\varepsilon)$ term to $\log(\Delta_i/\varepsilon)\log\log(1/\varepsilon)$, achieving log-log dependence when $\Delta_i$ and $\varepsilon$ are of the same order.
• The paper establishes non-asymptotic lower bounds showing that the combination of $\mu^{(\star)}$ and $\Delta$ is necessary for bounded regret, and that the proposed bounds are minimax optimal.