[Paper Review] Optimal Best-arm Identification in Linear Bandits

Yassir Jedra, Alexandre Proutière|arXiv (Cornell University)|Jun 29, 2020

Advanced Bandit Algorithms ResearchDecision Sciences24 references19 citations

TL;DR

This paper proposes a novel, scalable best-arm identification algorithm for stochastic linear bandits with finite and continuous arms, achieving asymptotically optimal sample complexity. The method uses a track-and-stop principle with a lazy arm sampling rule that tracks optimal draw proportions and a stopping rule independent of the number of arms, matching known information-theoretic lower bounds almost surely and in expectation.

ABSTRACT

We study the problem of best-arm identification with fixed confidence in stochastic linear bandits. The objective is to identify the best arm with a given level of certainty while minimizing the sampling budget. We devise a simple algorithm whose sampling complexity matches known instance-specific lower bounds, asymptotically almost surely and in expectation. The algorithm relies on an arm sampling rule that tracks an optimal proportion of arm draws, and that remarkably can be updated as rarely as we wish, without compromising its theoretical guarantees. Moreover, unlike existing best-arm identification strategies, our algorithm uses a stopping rule that does not depend on the number of arms. Experimental results suggest that our algorithm significantly outperforms existing algorithms. The paper further provides a first analysis of the best-arm identification problem in linear bandits with a continuous set of arms.

Motivation & Objective

• To address the lack of simple, scalable, and asymptotically optimal algorithms for best-arm identification in linear bandits with finite arms.
• To close the gap between existing algorithms and the instance-specific information-theoretic lower bound on expected sample complexity.
• To develop a method that does not depend on the number of arms in its stopping rule or sampling rule, enhancing scalability.
• To extend the analysis to the case of continuous arms, specifically the unit sphere, and derive a sample complexity lower bound.
• To propose an algorithm for continuous arms whose sample complexity matches the derived lower bound order-wise.

Proposed method

• The algorithm employs a track-and-stop strategy, where the sampling rule dynamically tracks the optimal proportion of draws for each arm based on the least-squares estimator of the parameter vector.
• The optimal proportion of draws is derived from the instance-specific sample complexity lower bound and updated rarely—enabling a 'lazy' implementation without sacrificing asymptotic optimality.
• The stopping rule uses a generalized likelihood ratio test with a novel exploration threshold that is independent of the number of arms.
• For continuous arms, the algorithm is designed for the unit sphere, and the stopping condition is constructed to ensure $(\epsilon,\delta)$-PAC guarantees.
• Theoretical guarantees are established via almost sure and expectation-based analysis, leveraging concentration inequalities and forced exploration to ensure convergence.
• The method's computational complexity can be tuned by adjusting the update frequency of the tracking rule, maintaining theoretical optimality.

Experimental results

Research questions

• RQ1 Can a best-arm identification algorithm in linear bandits achieve the information-theoretic lower bound on sample complexity while remaining simple and scalable?
• RQ2 Does a stopping rule independent of the number of arms exist that still ensures asymptotic optimality in best-arm identification?
• RQ3 What is the instance-specific sample complexity lower bound for best-arm identification in linear bandits with a continuous set of arms?
• RQ4 Can an algorithm be designed for continuous linear bandits whose sample complexity matches the derived lower bound order-wise?
• RQ5 How does the frequency of updating the optimal draw proportion tracking rule affect the algorithm's theoretical guarantees and computational cost?

Key findings

• The proposed algorithm achieves asymptotic optimality in sample complexity, matching the known instance-specific lower bound almost surely.
• The expected sample complexity of the algorithm is bounded by $\frac{d\sigma^2}{\varepsilon\|\mu\|}\log(1/\delta)$, matching the lower bound order-wise.
• The stopping rule is independent of the number of arms, enabling scalability to large or infinite arm sets.
• The tracking rule can be updated as infrequently as desired without affecting the asymptotic optimality, allowing for tunable computational cost.
• For continuous arms on the unit sphere, a sample complexity lower bound of order $\frac{d}{\varepsilon}\log(1/\delta)$ is established.
• An algorithm for continuous arms is proposed whose sample complexity matches this lower bound order-wise.