[Paper Review] Minimal Exploration in Structured Stochastic Bandits

Richard Combes, Stefan Magureanu+1 more|arXiv (Cornell University)|Nov 1, 2017

Advanced Bandit Algorithms Research27 references80 citations

TL;DR

The paper derives an instance-specific regret lower bound for a broad class of structured stochastic bandits and introduces OSSB, an asymptotically optimal algorithm that achieves this bound by matching minimal exploration rates of sub-optimal arms; experiments in linear bandits show OSSB outperforming Thompson sampling and related methods.

ABSTRACT

This paper introduces and addresses a wide class of stochastic bandit problems where the function mapping the arm to the corresponding reward exhibits some known structural properties. Most existing structures (e.g. linear, Lipschitz, unimodal, combinatorial, dueling, ...) are covered by our framework. We derive an asymptotic instance-specific regret lower bound for these problems, and develop OSSB, an algorithm whose regret matches this fundamental limit. OSSB is not based on the classical principle of "optimism in the face of uncertainty" or on Thompson sampling, and rather aims at matching the minimal exploration rates of sub-optimal arms as characterized in the derivation of the regret lower bound. We illustrate the efficiency of OSSB using numerical experiments in the case of the linear bandit problem and show that OSSB outperforms existing algorithms, including Thompson sampling.

Motivation & Objective

• Motivate learning in stochastic bandits with known structure in the arm-to-reward mapping.
• Derive an instance-specific asymptotic regret lower bound for structured bandits.
• Develop OSSB, an algorithm that matches the fundamental learning limits.
• Provide finite-time regret analysis and numerical evidence of OSSB’s efficiency.

Proposed method

• Model a generic structured MAB with unknown parameter e and structured reward mapping (x,).
• Derive a semi-infinite LP-based lower bound on sub-optimal exploration rates (Theorem 1).
• Define OSSB (Optimal Sampling for Structured Bandits) with exploitation, estimation, and exploration phases.
• Solve the lower-bound optimization problem (2)-(3) to guide exploration rates.
• Prove finite-time and asymptotic regret bounds for OSSB under Bernoulli or Gaussian rewards.
• Provide numerical experiments in linear bandits showing performance gains over TS and GLM-UCB.

Experimental results

Research questions

• RQ1 What is the minimal exploration rate required for each sub-optimal arm given the problem structure?
• RQ2 Can we design an algorithm that achieves the instance-specific lower bound for general structured bandits?
• RQ3 How does an algorithm that explicitly targets minimal exploration rates compare to optimism-based or Thompson sampling methods?
• RQ4 What are the finite-time regret guarantees of such an algorithm, and how do they translate to practical performance in linear bandits?

Key findings

• A tight instance-specific regret lower bound C() is derived via a semi-infinite LP, constraining exploration across sub-optimal arms.
• OSSB matches the lower bound asymptotically, by sampling sub-optimal arms at rates c(x,) log T.
• OSSB alternates between exploitation, estimation, and exploration to learn  while matching the optimal exploration rates.
• For linear bandits, OSSB outperforms Thompson sampling, GLM-UCB, and a recent asymptotically optimal algorithm in simulations.
• The algorithm is not based on optimism or Thompson sampling, and its analysis relies on concentration arguments and continuity of the optimal solution.
• Finite-time regret bounds are established, aligning with the asymptotic lower bound as parameters controlling exploration vanish.