[Paper Review] A/B Testing of Auctions

Shuchi Chawla, Jason D. Hartline+1 more|arXiv (Cornell University)|Jun 2, 2016

Auction Theory and Applications19 references4 citations

TL;DR

This paper develops a novel A/B testing methodology for auctions that enables accurate revenue estimation in settings where bidders are unaware of which mechanism (A or B) they are bidding under, avoiding interference from mixed-equilibrium behavior. By modeling the composite mechanism as a convex combination of A and B, the authors derive an estimator with error bounds that scale as log(1/ε) rather than 1/√ε, enabling reliable comparison even when B is rarely triggered (ε < 1/N).

ABSTRACT

For many application areas A/B testing, which partitions users of a system into an A (control) and B (treatment) group to experiment between several application designs, enables Internet companies to optimize their services to the behavioral patterns of their users. Unfortunately, the A/B testing framework cannot be applied in a straightforward manner to applications like auctions where the users (a.k.a., bidders) submit bids before the partitioning into the A and B groups is made. This paper combines auction theoretic modeling with the A/B testing framework to develop methodology for A/B testing auctions. The accuracy of our method %, assuming the auction is directly comparable to ideal A/B testing where there is no interference between A and B. Our results are based on an extension and improved analysis of the inference method of Chawla et al. (2014).

Motivation & Objective

• To address the fundamental challenge in A/B testing auctions where bidders are unaware of the mechanism they face, leading to mixed-equilibrium behavior that invalidates standard A/B comparisons.
• To develop a statistical estimator that accurately recovers the revenue of a target auction mechanism (e.g., B) from bids submitted under a composite mechanism (C) that mixes A and B.
• To improve upon prior error bounds by showing that estimation error depends on log(1/ε) rather than 1/√ε, enabling reliable inference even when the treatment mechanism B is rarely used.
• To demonstrate that comparing mechanisms (e.g., whether P_B1 > αP_B2) is statistically easier than estimating absolute revenue, with error rates decreasing exponentially in sample size.
• To enable practical optimization of auction layouts (e.g., number of mainline vs. sidebar ads) by allowing per-query, mechanism-specific A/B testing using a single composite mechanism.

Proposed method

• Model the composite A/B test mechanism C as a convex combination of the incumbent A and treatment B mechanisms, with B activated with probability ε.
• Use a weighted order statistic estimator based on equilibrium bid distributions from C to estimate the revenue of a target mechanism B, leveraging the mathematical equivalence of position auctions with different weights.
• Apply a refined analysis of the derivative of the equilibrium bid function to bound the error in estimating the revenue of a k-unit auction from bids in a composite mechanism C.
• Derive a key error bound: E[|P̂_k - P_k|] ≤ (40 / √N) × sup_q |x'_k(q)| × (n + log(1/ε)), where n is the number of positions and ε is the probability of B in C.
• For comparing mechanisms, construct a binary classifier γ = 1{P_B1 > αP_B2} and show its error rate decays exponentially with N, the number of bids, under fixed α and revenue gap.
• Adapt the analysis to multi-mechanism comparisons (e.g., selecting the best among r mechanisms), showing error bounds scale as r × exp(−O(Na² / (n³ log(rn/ε)))) with a being the gap between top two revenues.

Experimental results

Research questions

• RQ1 How can we accurately estimate the revenue of a target auction mechanism B when bids are collected under a composite mechanism C that mixes A and B?
• RQ2 What is the dependence of estimation error on the probability ε that mechanism B is used in the composite test C?
• RQ3 Can we compare the relative performance of two mechanisms (e.g., B1 vs. B2) more accurately than estimating their absolute revenues, and if so, how does the error scale with sample size?
• RQ4 Under what conditions can we reliably estimate multi-unit or position auction revenues from bids collected under a different mechanism?
• RQ5 How does the proposed method improve upon ideal A/B testing when the treatment mechanism is rarely triggered (ε < 1/N)?

Key findings

• The proposed estimator achieves error bounds that scale as log(1/ε) rather than 1/√ε, significantly improving robustness when the treatment mechanism is rarely used.
• For estimating the revenue of a k-unit auction from bids in a composite mechanism C = (1−ε)A + εB, the expected absolute error is bounded by (40 / √N) × sup_q |x'_k(q)| × (n + log(1/ε)).
• When comparing two mechanisms B1 and B2 via the classifier γ = 1{P_B1 > αP_B2}, the error rate decays exponentially with sample size N, specifically bounded by exp(−O(Na² / (α²n³ log(n/ε)))), where a = |P_B1 − αP_B2|.
• The method enables simultaneous estimation of all multi-unit and position auction revenues from a single set of bids collected under a composite mechanism C.
• For selecting the best among r mechanisms, the error probability is bounded by r × exp(−O(Na² / (n³ log(rn/ε)))), showing exponential convergence in N even with multiple comparisons.
• The method is especially effective in practical settings like ad layout optimization, where the number of mainline and sidebar ads (affecting click weights) can be tested per-query using the same composite mechanism.