[Paper Review] Randomization and The Pernicious Effects of Limited Budgets on Auction Experiments

Guillaume Basse, Hossein Azari Soufiani+1 more|arXiv (Cornell University)|May 30, 2016

Auction Theory and Applications11 references9 citations

TL;DR

This paper demonstrates that (query, advertiser) randomization in auction experiments introduces severe bias in treatment effect estimation due to interference between treated and control bidders, especially under joint budget throttling. It shows that query randomization and split-budget allocation significantly reduce bias, enabling unbiased estimation of bid and throttling treatment effects in ad auction experiments.

ABSTRACT

Buyers (e.g., advertisers) often have limited financial and processing resources, and so their participation in auctions is throttled. Changes to auctions may affect bids or throttling and any change may affect what winners pay. This paper shows that if an A/B experiment affects only bids, then the observed treatment effect is unbiased when all the bidders in an auction are randomly assigned to A or B but it can be severely biased otherwise, even in the absence of throttling. Experiments that affect throttling algorithms can also be badly biased, but the bias can be substantially reduced if the budget for each advertiser in the experiment is allocated to separate pots for the A and B arms of the experiment.

Motivation & Objective

• To analyze the impact of randomization schemes on bias in A/B experiments for ad auctions.
• To identify how budget constraints and throttling mechanisms distort treatment effect estimates.
• To compare the performance of query randomization versus (query, advertiser) randomization in terms of bias and variance.
• To evaluate the effectiveness of split versus joint budget allocation in reducing estimation bias.
• To provide practical guidelines for designing unbiased auction experiments under real-world budget and processing constraints.

Proposed method

• The authors model auction experiments using potential outcomes, defining potential bids and payments under treatment and control conditions for each advertiser.
• They introduce two randomization schemes: query randomization (entire auction assigned to one arm) and (query, advertiser) randomization (advertisers independently assigned).
• They formalize interference effects through causal models, particularly focusing on how competition between treated and control bidders distorts observed outcomes.
• They define joint and split quota systems, where joint quotas pool all bids under one budget, while split quotas allocate separate budgets per treatment arm.
• Simulations are conducted using lognormal bid distributions and varying throttling rates to assess bias and variance across different experimental designs.
• The study evaluates treatment effect estimation using relative bias and root mean squared error (RMSE), comparing estimators under different randomization and quota schemes.

Experimental results

Research questions

• RQ1 How does (query, advertiser) randomization introduce bias in estimating treatment effects for bid changes in ad auctions?
• RQ2 What is the impact of joint versus split budget allocation on the bias of treatment effect estimators in throttling experiments?
• RQ3 How does interference between treated and control bidders affect the validity of A/B test results in auction settings?
• RQ4 Does query randomization yield more accurate treatment effect estimates than (query, advertiser) randomization, and under what conditions?
• RQ5 Can split-budget allocation eliminate the severe bias observed in joint-budget throttling experiments?

Key findings

• The relative bias of treatment effect estimates under (query, advertiser) randomization can be as high as 1.5, indicating a 50% increase in bias compared to query randomization.
• For bid treatments, the variance of the estimator under (query, advertiser) randomization is up to 6 times higher than under query randomization when treatment bids are 5% higher than control bids and throttling is around 66%.
• Under joint quota throttling, the estimator for total revenue shows a relative bias of approximately -1, meaning it consistently estimates zero regardless of the true treatment effect.
• Split quota allocation reduces bias significantly, with relative bias close to zero even when theoretical conditions are violated, making it a robust alternative to joint quotas.
• The ratio of variances between (query, advertiser) and query randomization remains above one and may increase with sample size, indicating worse precision under the former.
• Simulations suggest that relative bias and variance ratios stabilize with larger sample sizes, implying that findings from small-scale simulations are representative of large-scale real-world experiments.