[Paper Review] Selecting the best system, large deviations, and multi-armed bandits

Peter W. Glynn, Sandeep Juneja|arXiv (Cornell University)|Jul 16, 2015

Advanced Bandit Algorithms Research23 references3 citations

TL;DR

This paper challenges the assumption that large deviations rate estimation enables O(log(1/δ)) sample complexity for identifying the best population with 1−δ confidence. It proves that under unbounded support, such guarantees are fundamentally impossible without moment restrictions, and shows that the estimated rate function itself follows a large deviations principle, offering new theoretical insight into simulation-based selection algorithms.

ABSTRACT

Consider the problem of finding a population amongst many with the largest mean when these means are unknown but population samples can be generated via simulation. Typically, by selecting a population with the largest sample mean, it can be shown that the false selection probability decays at an exponential rate. Lately researchers have sought algorithms that guarantee that this probability is restricted to a small $\delta$ in order $\log(1/\delta)$ computational time by estimating the associated large deviations rate function via simulation. We show that such guarantees are misleading. Enroute, we identify the large deviations principle followed by the empirically estimated large deviations rate function that may also be of independent interest. Further, we show a negative result that when populations have unbounded support, under mild restrictions, any policy that asymptotically identifies the correct population with probability at least $1-\delta$ for each problem instance requires more than $O(\log(1/\delta))$ samples in making such a determination in any problem instance. This suggests that some restrictions are essential on populations to devise $O(\log(1/\delta))$ algorithms with $1 - \delta$ correctness guarantees. We note that under restriction on population moments, such methods are easily designed. We also observe that sequential methods from stochastic multi-armed bandit literature can be adapted to devise such algorithms.

Motivation & Objective

• To investigate whether O(log(1/δ)) sample complexity is achievable for selecting the best population with 1−δ confidence using large deviations rate estimation.
• To analyze the large deviations behavior of the empirically estimated large deviations rate function.
• To determine whether moment restrictions are necessary for such efficient algorithms to exist.
• To evaluate the feasibility of sequential multi-armed bandit methods in achieving 1−δ correctness with O(log(1/δ)) samples.

Proposed method

• Derives the large deviations principle followed by the empirically estimated large deviations rate function using large deviations theory.
• Analyzes the asymptotic behavior of selection algorithms under unbounded population support.
• Establishes a negative result: any policy achieving 1−δ correctness for all instances requires more than O(log(1/δ)) samples when supports are unbounded.
• Uses mild moment restrictions to show that O(log(1/δ)) algorithms are possible under such conditions.
• Adapts sequential stochastic multi-armed bandit strategies to construct valid O(log(1/δ)) algorithms under moment constraints.
• Employs simulation-based estimation of large deviations rate functions to evaluate selection performance and error decay rates.

Experimental results

Research questions

• RQ1 Can large deviations rate estimation via simulation guarantee O(log(1/δ)) sample complexity for selecting the best population with 1−δ confidence?
• RQ2 What large deviations principle governs the empirically estimated large deviations rate function?
• RQ3 Is O(log(1/δ)) sample complexity fundamentally impossible when populations have unbounded support?
• RQ4 Are moment restrictions necessary to achieve O(log(1/δ)) algorithms with 1−δ correctness guarantees?
• RQ5 Can existing sequential multi-armed bandit methods be adapted to achieve such guarantees under moment constraints?

Key findings

• Under unbounded population support and mild restrictions, no policy can achieve 1−δ correctness with O(log(1/δ)) samples in all problem instances.
• The empirically estimated large deviations rate function follows its own large deviations principle, which is of independent theoretical interest.
• When populations have bounded moments, O(log(1/δ)) algorithms with 1−δ correctness guarantees can be constructed.
• The false selection probability decays exponentially under standard sampling, but this does not imply O(log(1/δ)) sample complexity for guaranteed confidence.
• Sequential multi-armed bandit methods can be adapted to achieve O(log(1/δ)) sample complexity under moment restrictions.
• The paper reveals that existing large deviations-based approaches may be misleading in practice due to fundamental limitations under unbounded support.