[Paper Review] Batched Multi-armed Bandits Problem

Zijun Gao, Yanjun Han+2 more|arXiv (Cornell University)|Apr 3, 2019

Advanced Bandit Algorithms Research34 references42 citations

TL;DR

This paper introduces BaSE, a batched successive elimination policy that achieves near-optimal minimax and problem-dependent regrets for batched MAB, with matching lower bounds for static and adaptive grids.

ABSTRACT

In this paper, we study the multi-armed bandit problem in the batched setting where the employed policy must split data into a small number of batches. While the minimax regret for the two-armed stochastic bandits has been completely characterized in \cite{perchet2016batched}, the effect of the number of arms on the regret for the multi-armed case is still open. Moreover, the question whether adaptively chosen batch sizes will help to reduce the regret also remains underexplored. In this paper, we propose the BaSE (batched successive elimination) policy to achieve the rate-optimal regrets (within logarithmic factors) for batched multi-armed bandits, with matching lower bounds even if the batch sizes are determined in an adaptive manner.

Motivation & Objective

• Motivate learning under limited interaction rounds where data arrives in batches.
• Characterize minimax and problem-dependent regret as a function of K arms, M batches, and horizon T.
• Develop policy(s) that attain rate-optimal regret within polylog factors under batched constraints.

Proposed method

• Propose Batched Successive Elimination (BaSE) based on exploring in the first M-1 batches and committing in the last batch.
• Use active-arm elimination with gap-dependent confidence bounds to remove clearly suboptimal arms at batch ends.
• Provide two static grids (minimax and geometric) to prove upper bounds; analyze regret bounds under those grids.
• Show that BaSE achieves upper bounds matching (up to polylog factors) the known fully adaptive rates when M grows with T.
• Derive lower bounds for static grids and general adaptive grids to establish minimax and problem-dependent limits.

Experimental results

Research questions

• RQ1 How does the number of batches M affect minimax and problem-dependent regret in K-armed batched bandits?
• RQ2 Can a batched policy approach the fully adaptive regret rates, and what grids (static/adaptive) are necessary to attain them?
• RQ3 What are the fundamental lower bounds for batched MAB under static and adaptive grids?
• RQ4 Do adaptive batch sizes provide a meaningful improvement over fixed grids in this setting?

Key findings

• For any K≥2, T≥1, 1≤M≤T, there exist BaSE policies achieving E[R_T] ≤ polylog(K,T) · sqrt(K) · T^{1/(2−2^{1−M})} (minimax grid).
• For the same setting, BaSE achieves E[R_T] ≤ polylog(K,T) · (K T^{1/M}) / min_{i≠*} Δ_i (geometric grid).
• Corollary: M = O(log log T) batches suffice for minimax regret Θ(√(K T)) and M = O(log T) batches suffice for problem-dependent regret Θ(K log T) (up to logarithmic factors).
• Lower bounds for static grids give R_min−max ≥ c √K · T^{1/(2−2^{1−M})} and R_pro−dep ≥ c K · T^{1/M}.
• Adaptive grids incur a polynomial M^{-2} factor in the lower bounds, yielding R_min−max ≥ c M^{-2} √K · T^{1/(2−2^{1−M})} and R_pro−dep ≥ c M^{-2} K · T^{1/M} (still polylogarithmically close to static-grid bounds).
• Corollaries: Ω(log log T) batches are necessary for minimax optimality and Ω(log T / log log T) for problem-dependent optimality, under either grid type.