[Paper Review] Optimizing Adaptive Experiments: A Unified Approach to Regret Minimization and Best-Arm Identification

Chao Qin, Daniel Russo|arXiv (Cornell University)|Feb 16, 2024

Advanced Statistical Process MonitoringDecision Sciences3 citations

TL;DR

This paper unifies regret minimization and best-arm identification in adaptive experiments by introducing a generalized bandit model with distinct costs for experimentation and deployment phases. It establishes that asymptotically optimal policies depend only on information balance for exploration and a tunable exploitation rate, enabling algorithms like top-two Thompson sampling to be adapted to a broad class of objectives via a single scalar parameter, with minimal regret even under drastic experiment length reductions.

ABSTRACT

Practitioners conducting adaptive experiments often encounter two competing priorities: maximizing total welfare (or `reward') through effective treatment assignment and swiftly concluding experiments to implement population-wide treatments. Current literature addresses these priorities separately, with regret minimization studies focusing on the former and best-arm identification research on the latter. This paper bridges this divide by proposing a unified model that simultaneously accounts for within-experiment performance and post-experiment outcomes. We provide a sharp theory of optimal performance in large populations that not only unifies canonical results in the literature but also uncovers novel insights. Our theory reveals that familiar algorithms, such as the recently proposed top-two Thompson sampling algorithm, can optimize a broad class of objectives if a single scalar parameter is appropriately adjusted. In addition, we demonstrate that substantial reductions in experiment duration can often be achieved with minimal impact on both within-experiment and post-experiment regret.

Motivation & Objective

• To bridge the gap between regret minimization and best-arm identification in adaptive experiments, which are currently treated as separate problems in the literature.
• To develop a unified theoretical framework that accounts for both within-experiment treatment costs and post-experiment deployment outcomes.
• To characterize the asymptotic trade-off between experiment length and total regret in large populations.
• To identify the structural properties of asymptotically efficient policies that are independent of per-period cost functions.
• To demonstrate that substantial reductions in experiment duration are possible with negligible impact on cumulative regret.

Proposed method

• Proposes a generalized multi-armed bandit model where treatment costs during experimentation and deployment are modeled separately via distinct per-period cost functions.
• Introduces a normalized performance metric scaling total cost logarithmically with population size, analogous to Lai and Robbins (1985) for regret minimization.
• Identifies that optimal exploration allocation follows an information-balance principle—evidence against suboptimal arms must grow at equal rates—making it independent of cost functions.
• Shows that cost-aware optimization reduces to tuning a single exploitation rate parameter, while exploration distribution is governed purely by statistical balance.
• Uses a two-player zero-sum game formulation to re-express the best-arm identification complexity constant in a form analogous to Lai and Robbins (1985).
• Applies the theory to characterize the Pareto frontier between experiment length and total regret, deriving exact asymptotic trade-offs.

Experimental results

Research questions

• RQ1 How can regret minimization and best-arm identification be formally unified in a single adaptive experiment framework?
• RQ2 What structural properties govern asymptotically efficient policies in the presence of heterogeneous per-period costs?
• RQ3 To what extent can experiment duration be reduced without significantly increasing total regret?
• RQ4 Can existing bandit algorithms be adapted to optimize a broad class of objectives with minimal modification?
• RQ5 What is the precise asymptotic trade-off between experiment length and total regret in large populations?

Key findings

• The optimal complexity constant for best-arm identification is re-expressed as the value of a two-player zero-sum game, which matches the form of Lai and Robbins (1985) under a change of variables.
• Asymptotically efficient policies are nearly independent of cost functions; exploration effort should be allocated to balance statistical evidence against suboptimal arms.
• The top-two Thompson sampling algorithm is asymptotically efficient for Gaussian bandits, as it satisfies the information-balance condition.
• Significant reductions in experiment length are possible with minimal regret impact—up to 50% shorter experiments can be achieved with less than 5% increase in total regret in some regimes.
• The Pareto frontier between experiment length and total regret is exactly characterized, with the minimal normalized length and regret achieved by policies parameterized by β ∈ (β_BAI, 1).
• For Gaussian bandits, the normalized regret scales as R_θ^(β) ≤ (1/β) R_θ^(1), showing that regret increases linearly with 1/β as β approaches 1.