[Paper Review] Optimal $δ$-Correct Best-Arm Selection for Heavy-Tailed Distributions

Shubhada Agrawal, Sandeep Juneja+1 more|arXiv (Cornell University)|Aug 24, 2019

Machine Learning and Algorithms5 citations

TL;DR

This paper proposes a δ-correct best-arm identification algorithm for heavy-tailed distributions with finite (1+ε)th moments, achieving asymptotic optimality in sample complexity. It introduces batch processing with near-optimal batch sizes to accelerate convergence, overcoming limitations of prior methods restricted to exponential family distributions.

ABSTRACT

Given a finite set of unknown distributions or arms that can be sampled, we consider the problem of identifying the one with the maximum mean using a $δ$-correct algorithm (an adaptive, sequential algorithm that restricts the probability of error to a specified $δ$) that has minimum sample complexity. Lower bounds for $δ$-correct algorithms are well known. $δ$-correct algorithms that match the lower bound asymptotically as $δ$ reduces to zero have been previously developed when arm distributions are restricted to a single parameter exponential family. In this paper, we first observe a negative result that some restrictions are essential, as otherwise, under a $δ$-correct algorithm, distributions with unbounded support would require an infinite number of samples in expectation. We then propose a $δ$-correct algorithm that matches the lower bound as $δ$ reduces to zero under the mild restriction that a known bound on the expectation of $(1+ε)^{th}$ moment of the underlying random variables exists, for $ε> 0$. We also propose batch processing and identify near-optimal batch sizes to speed up the proposed algorithm substantially. The best-arm problem has many learning applications, including recommendation systems and product selection. It is also a well-studied classic problem in the simulation community.

Motivation & Objective

• Address the lack of optimal δ-correct algorithms for general heavy-tailed distributions beyond the exponential family.
• Establish that without moment restrictions, δ-correct algorithms require infinite expected samples, necessitating distributional assumptions.
• Propose a δ-correct algorithm matching the information-theoretic lower bound as δ→0 under a mild (1+ε)th moment condition.
• Develop batch processing techniques with near-optimal batch sizes to significantly speed up the algorithm.
• Provide a revised, tighter concentration analysis using the Lévy metric to correct technical errors from prior work.

Proposed method

• Use the Lévy metric instead of the Wasserstein metric to derive tighter concentration inequalities.
• Define the distribution class L as all distributions with bounded (1+ε)th absolute moments for ε>0.
• Design a sequential sampling algorithm that adaptively allocates pulls based on empirical means and confidence bounds.
• Introduce batch processing where sampling decisions are made in batches, reducing computational overhead.
• Optimize batch size proportional to log(1/δ) to balance exploration and convergence speed.
• Apply Azuma-Hoeffding inequality to bound deviations in sampling proportions across batches under martingale constraints.

Experimental results

Research questions

• RQ1 What distributional restrictions are necessary to ensure finite expected sample complexity in δ-correct best-arm identification?
• RQ2 Can a δ-correct algorithm achieve asymptotic optimality for heavy-tailed distributions with only finite (1+ε)th moments?
• RQ3 How can batch processing be designed to maintain asymptotic optimality while improving computational efficiency?
• RQ4 What are the consequences of using the Wasserstein metric versus the Lévy metric in concentration analysis for this problem?
• RQ5 How can technical errors in prior work—particularly in concentration bounds and measure change analysis—be corrected?

Key findings

• Any δ-correct algorithm for distributions with unbounded support and KL-right-dense properties requires infinite expected samples, establishing a fundamental impossibility without moment restrictions.
• The proposed algorithm achieves asymptotic optimality in sample complexity as δ→0 under the (1+ε)th moment condition, matching the information-theoretic lower bound.
• Batch processing with batch size proportional to log(1/δ) reduces computational cost while preserving asymptotic optimality.
• The use of the Lévy metric enables a tighter concentration inequality, correcting errors present in earlier work using the Wasserstein metric.
• The algorithm’s error probability is bounded by δ, and the expected sample complexity matches the lower bound up to a logarithmic factor.
• Numerical analysis confirms that the batched variant significantly reduces runtime without sacrificing δ-correctness or asymptotic optimality.