[Paper Review] Structured Best Arm Identification with Fixed Confidence

Ruitong Huang, Mohammad M. Ajallooeian+2 more|arXiv (Cornell University)|Jun 16, 2017

Advanced Bandit Algorithms Research4 references3 citations

TL;DR

This paper introduces a novel algorithm, LUCB-micro, for structured best arm identification in the fixed-confidence setting, where arm values are functions of noisy micro-observables. It provides instance-dependent sample complexity bounds that generalize and improve upon prior work, particularly in minimax game search with non-uniform depth and transpositions, achieving tighter bounds than existing methods like FindTopWinner.

ABSTRACT

We study the problem of identifying the best action among a set of possible options when the value of each action is given by a mapping from a number of noisy micro-observables in the so-called fixed confidence setting. Our main motivation is the application to the minimax game search, which has been a major topic of interest in artificial intelligence. In this paper we introduce an abstract setting to clearly describe the essential properties of the problem. While previous work only considered a two-move game tree search problem, our abstract setting can be applied to the general minimax games where the depth can be non-uniform and arbitrary, and transpositions are allowed. We introduce a new algorithm (LUCB-micro) for the abstract setting, and give its lower and upper sample complexity results. Our bounds recover some previous results, which were only available in more limited settings, while they also shed further light on how the structure of minimax problems influence sample complexity.

Motivation & Objective

• To address best arm identification in structured environments where arm payoffs depend on noisy micro-observables, extending beyond two-move game trees.
• To formalize an abstract setting that captures essential properties of minimax game search with arbitrary depth and transpositions.
• To develop a new algorithm, LUCB-micro, that efficiently identifies the best arm under fixed-confidence constraints.
• To establish tight instance-dependent sample complexity bounds that reflect the influence of problem structure on learning efficiency.
• To recover and improve upon prior results in two-move alternating games and extend them to full-length minimax trees.

Proposed method

• Propose an abstract framework where arm payoffs are deterministic functions of unknown micro-observable values, with noisy observations available per micro-observable.
• Introduce the LUCB-micro algorithm as a generalization of Maximin-LUCB, using upper and lower confidence bounds on micro-observables to guide sampling.
• Define proof sets as critical structures in minimax trees that determine the difficulty of identifying the optimal move, analogous to conspiracy sets.
• Use a min-max computation strategy to efficiently compute the best action choices in the minimax setting, leveraging the structure of the payoff function.
• Establish high-probability, instance-dependent upper bounds on sample complexity by analyzing the number of pulls per micro-observable based on confidence intervals and span of value sets.
• Prove that the algorithm meets the risk requirement (i.e., correct identification with high probability) under regularity assumptions on the payoff map.

Experimental results

Research questions

• RQ1 Can the efficiency gains observed in two-move game trees be extended to general minimax games with arbitrary depth and transpositions?
• RQ2 How does the structure of the game tree—particularly the presence of shared states and non-uniform depth—affect the sample complexity of best arm identification?
• RQ3 What is the instance-dependent lower bound on sample complexity in structured best arm identification with micro-observables?
• RQ4 Can a single algorithm like LUCB-micro achieve optimal sample complexity across diverse structured environments including full minimax trees?
• RQ5 How do proof sets, as structural invariants in minimax trees, influence the difficulty of identifying the optimal move?

Key findings

• The proposed LUCB-micro algorithm achieves a high-probability, instance-dependent upper bound on sample complexity that is strictly better than FindTopWinner in the minimax game setting, particularly in the number of samples required for near-optimal micro-observables.
• The sample complexity upper bound of LUCB-micro matches the instance-dependent lower bound in the two-move alternating case, confirming its optimality in that regime.
• The paper establishes a new instance-dependent lower bound for structured best arm identification, which generalizes and recovers prior results from Garivier et al. (2016a) in the two-move setting.
• The regularity assumptions required for the abstract setting are satisfied in the minimax game setting, enabling the application of the theoretical guarantees to real game search problems.
• The use of proof sets as structural invariants allows for tighter analysis of sample complexity, revealing that the difficulty of identifying the best move is governed by the minimal proof set size.
• The algorithm's sample complexity is shown to be proportional to the inverse square of the span of value sets over proof sets, demonstrating a clean dependence on the problem's intrinsic difficulty.