[Paper Review] Maximin Action Identification: A New Bandit Framework for Games

Aurélien Garivier, Emilie Kaufmann+1 more|arXiv (Cornell University)|Feb 15, 2016

Advanced Bandit Algorithms Research12 references3 citations

TL;DR

This paper introduces a novel bandit framework for identifying the maximin action in two-player zero-sum games, where a player samples random outcomes of action pairs to estimate win probabilities. It proposes two algorithms—Maximin-LUCB and Maximin-Racing—using confidence bounds and sequential elimination, respectively, achieving optimal sample complexity in theory and strong empirical performance, with theoretical and empirical evidence pointing toward asymptotic optimality under specific conditions.

ABSTRACT

We study an original problem of pure exploration in a strategic bandit model motivated by Monte Carlo Tree Search. It consists in identifying the best action in a game, when the player may sample random outcomes of sequentially chosen pairs of actions. We propose two strategies for the fixed-confidence setting: Maximin-LUCB, based on lower-and upper-confidence bounds; and Maximin-Racing, which operates by successively eliminating the sub-optimal actions. We discuss the sample complexity of both methods and compare their performance empirically. We sketch a lower bound analysis, and possible connections to an optimal algorithm.

Motivation & Objective

• To address the fundamental challenge of minimizing rollouts in Monte Carlo Tree Search (MCTS) by identifying the best strategic action under uncertainty.
• To formalize a new bandit model where the goal is to identify the action maximizing the minimum win probability against an adversarial opponent.
• To design and analyze dynamic sampling strategies that minimize the number of samples (rollouts) required to identify an ǫ-maximin action with high confidence.
• To establish theoretical sample complexity bounds and explore connections to asymptotic optimality in the fixed-confidence setting.

Proposed method

• Proposes a stochastic bandit model with K = ∑Ki Bernoulli arms, each corresponding to a pair of actions (i,j), where the mean µi,j is the win probability for player A when choosing action i and opponent choosing j.
• Introduces Maximin-LUCB, a strategy using lower- and upper-confidence bounds to sequentially sample arms and stop when the confidence interval on the maximin value of the best action is narrow enough.
• Develops Maximin-Racing, a sequential elimination algorithm that removes sub-optimal actions based on confidence intervals, ensuring δ-PAC performance.
• Employs a fixed-confidence setting where the stopping rule ensures the final recommendation ˆı is an ǫ-maximin action with probability at least 1−δ.
• Analyzes sample complexity using Hoeffding and KL-divergence bounds, and derives a non-explicit lower bound on the expected number of samples.
• Sketches a path toward asymptotic optimality by designing a sampling rule that asymptotically matches the optimal proportion vector w∗(µ), derived from a convex optimization problem over the simplex.

Experimental results

Research questions

• RQ1 What is the minimal number of samples (rollouts) required to identify an ǫ-maximin action in a two-player zero-sum game with unknown win probabilities?
• RQ2 How can sequential sampling strategies be designed to minimize sample complexity while ensuring δ-PAC correctness in this novel bandit framework?
• RQ3 What is the theoretical sample complexity lower bound for this maximin action identification problem?
• RQ4 Can the performance of existing algorithms like Maximin-LUCB and Maximin-Racing be improved by using tighter confidence bounds or more adaptive sampling rules?
• RQ5 Is there a strategy that asymptotically matches the information-theoretic lower bound as δ → 0?

Key findings

• Maximin-Racing and Maximin-Chernoff outperform other algorithms in empirical evaluations, with M-Chernoff showing a slight edge in sample efficiency.
• The M-LUCB algorithm, which uses Hoeffding bounds, is sub-optimal due to conservative confidence intervals, as confirmed by the empirical comparison with M-KL-LUCB and M-Chernoff.
• In a 3×3 action model, the number of draws for each arm under M-KL-LUCB and M-Chernoff closely follows the theoretical bounds, validating the algorithmic design.
• The theoretical lower bound on sample complexity is expressed as T∗(µ)−1 = supw∈ΣK min[F1(µ,w), F2(µ,w)], where F1 and F2 depend on KL-divergence and weighted averages of means.
• When µ4 > µ2, the optimal strategy asymptotically avoids sampling arm 4 (i.e., w∗4(µ) = 0), indicating that some arms are irrelevant in the optimal sampling strategy.
• The paper suggests that an asymptotically optimal algorithm could be constructed by combining a sampling rule that achieves the optimal proportion vector w∗(µ) with a stopping rule based on KL-divergence, though full proof remains open.