[Paper Review] A/B/n Testing with Control in the Presence of Subpopulations

Yoan Russac, Christina Katsimerou+4 more|arXiv (Cornell University)|Oct 29, 2021

Statistical Methods in Clinical Trials4 citations

TL;DR

This paper proposes an asymptotically optimal sequential A/B/n testing strategy for identifying all arms better than a control in the presence of subpopulations, using adaptive sampling based on subpopulation information. The method achieves optimal sample complexity by dynamically allocating samples to arms based on uncertainty, with theoretical guarantees on stopping time and error rate, and is effective across three interaction modes: active, passive, and oblivious sampling of subpopulations.

ABSTRACT

Motivated by A/B/n testing applications, we consider a finite set of distributions (called \emph{arms}), one of which is treated as a \emph{control}. We assume that the population is stratified into homogeneous subpopulations. At every time step, a subpopulation is sampled and an arm is chosen: the resulting observation is an independent draw from the arm conditioned on the subpopulation. The quality of each arm is assessed through a weighted combination of its subpopulation means. We propose a strategy for sequentially choosing one arm per time step so as to discover as fast as possible which arms, if any, have higher weighted expectation than the control. This strategy is shown to be asymptotically optimal in the following sense: if $τ_δ$ is the first time when the strategy ensures that it is able to output the correct answer with probability at least $1-δ$, then $\mathbb{E}[τ_δ]$ grows linearly with $\log(1/δ)$ at the exact optimal rate. This rate is identified in the paper in three different settings: (1) when the experimenter does not observe the subpopulation information, (2) when the subpopulation of each sample is observed but not chosen, and (3) when the experimenter can select the subpopulation from which each response is sampled. We illustrate the efficiency of the proposed strategy with numerical simulations on synthetic and real data collected from an A/B/n experiment.

Motivation & Objective

• To develop a sequential testing policy that efficiently identifies all arms better than a control in the presence of subpopulations.
• To address the inefficiency of uniform sampling in A/B/n testing by adapting sample allocation based on subpopulation structure.
• To provide theoretical guarantees on sample complexity and error probability under varying levels of subpopulation observability and control.
• To enable anytime decision-making with calibrated risk assessments, supporting early stopping in practical deployments.
• To evaluate the impact of subpopulation interaction modes—active, proportional, agnostic, and oblivious—on decision speed and accuracy.

Proposed method

• Proposes a Track-and-Stop framework adapted for ABC-S (Arms Better than Control in Subpopulations), using sequential sampling and stopping rules.
• Introduces a weighted expectation criterion that combines subpopulation means according to their frequency and importance.
• Employs a confidence-bound-based sampling rule that prioritizes arms with highest uncertainty relative to the control, minimizing expected sample complexity.
• Designs a risk assessment mechanism that updates sequentially and ensures the probability of incorrect recommendation is bounded by the reported risk.
• Considers three interaction modes: (1) active (learner selects subpopulation), (2) passive (subpopulation observed but not selected), and (3) oblivious (no subpopulation access).
• Derives asymptotic optimality by showing that expected stopping time E[τδ] grows linearly with log(1/δ) at the optimal rate across all three modes.

Experimental results

Research questions

• RQ1 Can a sequential sampling strategy achieve asymptotic optimality in identifying all arms better than the control when subpopulations are present?
• RQ2 How does the ability to select or observe subpopulations affect the sample complexity of identifying superior arms?
• RQ3 Can a risk assessment be maintained throughout the experiment to enable anytime stopping with guaranteed error control?
• RQ4 What is the theoretical sample complexity rate in the fixed-confidence setting under different subpopulation interaction modes?
• RQ5 How do adaptive sampling policies compare to uniform sampling in real and synthetic A/B/n experiments with subpopulations?

Key findings

• The proposed strategy achieves asymptotic optimality, with E[τδ] growing at the optimal rate of log(1/δ) across all three interaction modes.
• Active sampling, where the learner selects the subpopulation, reduces decision time significantly compared to passive or oblivious modes.
• In the real-world dataset with seasonal subpopulations, adaptive sampling reduced sample complexity compared to uniform sampling, with the active mode terminating fastest.
• The risk assessment mechanism correctly bounds the probability of incorrect recommendation, with uniform sampling showing a high risk assessment (0.67) even when correct.
• Proportional and agnostic modes performed similarly, but proportional was recommended as it never underperforms on average.
• Even with temporal dependence in subpopulations (e.g., seasonal cycles), the method remains effective when cycles are frequently observed.