[Paper Review] A minimax and asymptotically optimal algorithm for stochastic bandits

Pierre Ménard, Aurélien Garivier|arXiv (Cornell University)|Feb 23, 2017

Advanced Bandit Algorithms Research72 citations

TL;DR

The paper introduces the kl-UC B++ algorithm for stochastic bandits with exponential-family distributions and proves it is both minimax- and asymptotically-optimal, unifying two time-optimality notions.

ABSTRACT

We propose the kl-UCB ++ algorithm for regret minimization in stochastic bandit models with exponential families of distributions. We prove that it is simultaneously asymptotically optimal (in the sense of Lai and Robbins' lower bound) and minimax optimal. This is the first algorithm proved to enjoy these two properties at the same time. This work thus merges two different lines of research with simple and clear proofs.

Motivation & Objective

• Motivate the need for an algorithm that is simultaneously minimax- and asymptotically-optimal in stochastic bandits.
• Develop and analyze the kl-UC B++ algorithm leveraging exponential-family distributions.
• Provide rigorous finite-time and asymptotic regret guarantees that reconcile two time-optimality notions.
• Offer a unified, simple proof framework with new deviation inequalities for bandit regret analysis.

Proposed method

• Define kl-UC B++ with an exploration function g(n) = log_plus( (T/(Kn)) (log_plus^2(T/(Kn)) + 1) ).
• Use upper confidence bounds U_a(t) defined via KL divergence constraints: U_a(t) = sup{ mu in I : kl( mu_hat_a(t), mu ) <= g(N_a(t))/N_a(t) }.
• Base analysis on exponential-family assumptions with bounded mean interval [mu-, mu+] and a uniform variance bound V.
• Prove minimax optimality: R_T ≤ 76 sqrt(V K T) + (mu^+ − mu^-) K.
• Prove asymptotic optimality: for each suboptimal arm a and suitable delta, E[N_a(T)] ≤ log(T)/kl(mu_a+delta, mu^*−delta) + lower-order terms.

Experimental results

Research questions

• RQ1 Can a single bandit algorithm achieve both minimax and problem-dependent (asymptotic) optimality for exponential-family bandits?
• RQ2 How should exploration be tuned to simultaneously minimize worst-case regret and achieve tight problem-dependent regret rates?
• RQ3 Do KL-based UCB strategies extend to provide uniform regret guarantees across a family of distributions with bounded variance?
• RQ4 What are the precise finite-time regret bounds and asymptotic characterizations for kl-UC B++ in Bernoulli/ Gaussian (and other) exponential families?

Key findings

• The kl-UC B++ algorithm achieves both minimax-optimal and asymptotically optimal regret for exponential-family bandits.
• A finite-time regret bound is established: R_T ≤ 76 sqrt(V K T) + (mu^+ − mu^-) K.
• For any suboptimal arm a and suitable delta, E[N_a(T)] ≤ log(T)/kl(mu_a+delta, mu^*−delta) + O(log log T / delta^2).
• The analysis covers Bernoulli and Gaussian cases as special instances within the exponential-family framework.
• The approach merges prior work on MOSS and KL-UCB by adjusting exploration rates, yielding a unified, simpler proof structure.