[Paper Review] Exploring $k$ out of Top $ρ$ Fraction of Arms in Stochastic Bandits

Wenbo Ren, Jia Liu+1 more|arXiv (Cornell University)|Oct 28, 2018

Advanced Bandit Algorithms Research7 citations

TL;DR

This paper introduces a novel quantile exploration (QE) framework for stochastic multi-armed bandits that identifies $k$ arms within the top $\rho$ fraction of expected rewards under PAC guarantees. It proposes algorithms for both known and unknown thresholds, achieving sample complexity optimal up to constant or logarithmic factors, with up to $\rho n/k$ reductions over traditional $k$-exploration methods.

ABSTRACT

This paper studies the problem of identifying any $k$ distinct arms among the top $ρ$ fraction (e.g., top 5\%) of arms from a finite or infinite set with a probably approximately correct (PAC) tolerance $ε$. We consider two cases: (i) when the threshold of the top arms' expected rewards is known and (ii) when it is unknown. We prove lower bounds for the four variants (finite or infinite arms, and known or unknown threshold), and propose algorithms for each. Two of these algorithms are shown to be sample complexity optimal (up to constant factors) and the other two are optimal up to a log factor. Results in this paper provide up to $ρn/k$ reductions compared with the "$k$-exploration" algorithms that focus on finding the (PAC) best $k$ arms out of $n$ arms. We also numerically show improvements over the state-of-the-art.

Motivation & Objective

• To address the limitation of traditional $k$-exploration in large or infinite arm sets by focusing on identifying arms within a top quantile rather than the absolute best $k$.
• To reduce sample complexity dependence on $n$ (total arms) by replacing it with $k/\rho$, enabling scalability in large-scale applications.
• To study the pure exploration problem under PAC tolerance $\epsilon$ for both finite and infinite arm sets with known and unknown thresholds.
• To derive theoretical lower bounds and design algorithms that match or nearly match these bounds in sample complexity.

Proposed method

• Proposes the Quantile Exploration (QE) problem as a relaxation of $k$-exploration, where the goal is to find $k$ arms in the top $\rho$ fraction of expected rewards.
• Introduces two algorithmic variants: CB-AL-Q-IK (for known threshold $\lambda_\rho$) and CB-AL-Q-IU (for unknown threshold), using confidence bounds and arm elimination.
• Employs a CBB (Confidence-Bound-Based) framework with adaptive sampling and pruning based on upper and lower confidence bounds.
• Uses inverse CDF $\mathcal{F}^{-1}(\rho)$ to define the threshold for the top $\rho$ fraction, enabling quantile-based arm selection.
• Applies a two-phase sampling strategy: first, estimate arm means with high confidence; second, eliminate arms below the quantile threshold.
• For unknown threshold, uses a sequential elimination procedure (LE) with confidence intervals to progressively narrow down candidates.

Experimental results

Research questions

• RQ1 Can we reduce sample complexity in pure exploration bandits by targeting arms within the top $\rho$ fraction instead of the top $k$ arms?
• RQ2 What are the theoretical lower bounds on sample complexity for identifying $k$ arms in the top $\rho$ fraction under known and unknown threshold settings?
• RQ3 How do the sample complexities of proposed algorithms compare to existing $k$-exploration methods in terms of dependence on $n$, $k$, $\rho$, and $\epsilon$?
• RQ4 Can we achieve sample complexity optimality (up to constant or logarithmic factors) for both finite and infinite arm sets in the QE framework?

Key findings

• The proposed CB-AL-Q-IK algorithm achieves sample complexity optimal up to constant factors for the known-threshold finite-arm case.
• For the unknown-threshold finite-arm case, CB-AL-Q-IU achieves sample complexity optimal up to a $\log k$ factor.
• The sample complexity of the QE problem is reduced by up to $\rho n/k$ compared to traditional $k$-exploration, especially beneficial when $\rho$ is small.
• Numerical results show CB-AL-Q-IK outperforms KL-LUCB in all tested settings, with significantly lower sample counts, especially when $\rho$ is small.
• The algorithm’s sample count is nearly independent of $\delta$ when $\rho$ is small, due to the $\frac{1}{\rho}\log\frac{1}{\rho}$ term dominating $\log\frac{1}{\delta}$.
• For large $k$, CB-AL-Q-IU shows a clear advantage over KL-LUCB due to its $k\log k$ dependence versus KL-LUCB’s $k\log^2 k$ scaling.