[Paper Review] PAC Identification of Many Good Arms in Stochastic Multi-Armed Bandits

Arghya Roy Chaudhuri, Shivaram Kalyanakrishnan|arXiv (Cornell University)|Jan 24, 2019

Advanced Bandit Algorithms Research28 references3 citations

TL;DR

This paper introduces the PAC identification problem of selecting $k$ out of the best $m$ arms in stochastic multi-armed bandits, generalizing both best-subset selection and single-arm identification. It proposes the LUCB-$k$-$m$ algorithm, which achieves sample complexity matching known lower bounds up to constant factors, and extends the framework to infinite-armed bandits with poly-logarithmic gap to optimality.

ABSTRACT

We consider the problem of identifying any $k$ out of the best $m$ arms in an $n$-armed stochastic multi-armed bandit. Framed in the PAC setting, this particular problem generalises both the problem of `best subset selection' and that of selecting `one out of the best m' arms [arcsk 2017]. In applications such as crowd-sourcing and drug-designing, identifying a single good solution is often not sufficient. Moreover, finding the best subset might be hard due to the presence of many indistinguishably close solutions. Our generalisation of identifying exactly $k$ arms out of the best $m$, where $1 \leq k \leq m$, serves as a more effective alternative. We present a lower bound on the worst-case sample complexity for general $k$, and a fully sequential PAC algorithm, \GLUCB, which is more sample-efficient on easy instances. Also, extending our analysis to infinite-armed bandits, we present a PAC algorithm that is independent of $n$, which identifies an arm from the best $ρ$ fraction of arms using at most an additive poly-log number of samples than compared to the lower bound, thereby improving over [arcsk 2017] and [Aziz+AKA:2018]. The problem of identifying $k > 1$ distinct arms from the best $ρ$ fraction is not always well-defined; for a special class of this problem, we present lower and upper bounds. Finally, through a reduction, we establish a relation between upper bounds for the `one out of the best $ρ$' problem for infinite instances and the `one out of the best $m$' problem for finite instances. We conjecture that it is more efficient to solve `small' finite instances using the latter formulation, rather than going through the former.

Motivation & Objective

• To address practical scenarios where identifying a single optimal arm is insufficient, such as distributed crowd-sourcing or drug design, requiring multiple good solutions.
• To generalize existing problems: best-subset selection ($k=m$) and one-out-of-best-$m$ ($k=1$) into a unified framework for identifying $k$ arms from the top $m$ among $n$ arms.
• To establish theoretical sample complexity bounds for the generalized $(k,m,n)$ problem in both finite and infinite-armed bandit settings.
• To design a fully sequential, adaptive PAC algorithm, LUCB-$k$-$m$, that efficiently identifies $k$ good arms with minimal sampling.
• To analyze the feasibility and complexity of identifying $k$ distinct arms from the best $\rho$ fraction in infinite-armed bandits, and to relate finite and infinite instances via reduction.

Proposed method

• Proposes a new problem formulation: identify $k$ arms from the best $m$ among $n$ arms in a stochastic multi-armed bandit under the PAC setting.
• Introduces the LUCB-$k$-$m$ algorithm, a fully sequential, adaptive algorithm that uses upper and lower confidence bounds to sequentially eliminate suboptimal arms.
• Derives a lower bound on the worst-case sample complexity for the $(k,m,n)$ problem, generalizing prior bounds for $k=1$ and $k=m$.
• Extends the analysis to infinite-armed bandits by defining $\rho$-optimal arms (within $\epsilon$ of the $\rho$-quantile of means), and presents a PAC algorithm with sample complexity within an additive poly-log term of the lower bound.
• Identifies a class of well-posed instances for $k$-out-of-best-$\rho$-fraction problems and provides a corresponding algorithm with theoretical guarantees.
• Establishes a reduction between the infinite-armed $(k,\rho)$-problem and the finite-armed $(k,m,n)$-problem, suggesting that solving the finite version may be more efficient.

Experimental results

Research questions

• RQ1 What is the fundamental sample complexity lower bound for identifying $k$ arms from the best $m$ among $n$ arms in a stochastic bandit?
• RQ2 Can a fully sequential, adaptive algorithm be designed that achieves near-optimal sample complexity for the generalized $(k,m,n)$ problem?
• RQ3 How does the sample complexity of identifying $k$ arms from the best $\rho$ fraction scale in infinite-armed bandits, and can it be made close to the information-theoretic lower bound?
• RQ4 Under what conditions is the problem of identifying $k$ distinct $[\epsilon,\rho]$-optimal arms well-posed, and how can it be solved efficiently?
• RQ5 Is there a structural advantage in solving the finite-armed $(k,m,n)$ problem over the infinite-armed $(k,\rho)$ problem in terms of sample efficiency?

Key findings

• The LUCB-$k$-$m$ algorithm achieves expected sample complexity that matches known lower bounds for the $k=1$ and $k=m$ cases up to a constant factor.
• Empirical evaluation shows LUCB-$k$-$m$ significantly outperforms existing algorithms like $\mathcal{F}_2$ in terms of sample efficiency, especially as $n$ grows.
• For infinite-armed bandits, the proposed algorithm identifies a $[\epsilon,\rho]$-optimal arm with sample complexity within an additive $O(\log^2(1/\delta))$ term of the lower bound, improving upon prior work.
• The problem of identifying $k$ distinct $[\epsilon,\rho]$-optimal arms is not always well-defined, but a specific class of instances admits a well-posed formulation with provable bounds.
• A reduction is established between the infinite-armed $(k,\rho)$-problem and the finite-armed $(k,m,n)$-problem, suggesting that finite instances may be more efficiently solvable.
• The authors conjecture that solving the finite-armed $(k,m,n)$ problem is more sample-efficient than solving the corresponding infinite-armed $(k,\rho)$ problem, a direction left open for future work.