[Paper Review] Principal Stratification for Advertising Experiments

Ron Berman, Elea McDonnell Feit|arXiv (Cornell University)|Nov 19, 2019

Consumer Market Behavior and PricingBusiness, Management and Accounting25 references3 citations

TL;DR

This paper proposes a principal stratification model that improves precision in advertising experiment ATE estimation by stratifying customers into three groups: always buyers, defiers (buy only if exposed), and never buyers. By isolating the large group of always buyers—whose treatment effects are exactly zero—the method reduces ATE variance by 36–57% in five catalog mailing experiments with ~140,000 participants.

ABSTRACT

Advertising experiments often suffer from noisy responses making precise estimation of the average treatment effect (ATE) and evaluating ROI difficult. We develop a principal stratification model that improves the precision of the ATE by dividing the customers into three strata - those who buy regardless of ad exposure, those who buy only if exposed to ads and those who do not buy regardless. The method decreases the variance of the ATE by separating out the typically large share of customers who buy and therefore have individual treatment effects that are exactly zero. Applying the procedure to 5 catalog mailing experiments with sample sizes around 140,000 shows a reduction of 36-57% in the variance of the estimate. When we include pre-randomization covariates that predict stratum membership, we find that estimates of customers' past response to similar advertising are a good predictor of stratum membership, even if such estimates are biased because past advertising was targeted. Customers who have not purchased recently are also more likely to be in the never purchase stratum. We provide simple summary statistics that firms can compute from their own experiment data to determine if the procedure is expected to be beneficial before applying it.

Motivation & Objective

• To address the challenge of noisy responses in advertising experiments that hinder precise estimation of the average treatment effect (ATE).
• To reduce the variance of ATE estimates by accounting for heterogeneous customer response behaviors in a structured way.
• To evaluate whether pre-randomization covariates—such as past response to advertising—can predict stratum membership and improve estimation.
• To provide practical summary statistics that firms can use to assess whether the method will be beneficial before implementation.

Proposed method

• Stratifies customers into three principal strata based on potential outcomes: always buyers (always purchase regardless of ad exposure), defiers (purchase only if exposed), and never buyers (never purchase regardless of exposure).
• Models the ATE within each stratum and combines estimates to produce a more precise overall ATE by reducing variance from the inclusion of zero-effect units.
• Uses pre-randomization covariates—such as past purchase behavior—to predict stratum membership, even when past advertising was targeted and thus biased.
• Applies the stratification model to five real-world catalog mailing experiments with sample sizes around 140,000 to evaluate variance reduction.
• Employs simple summary statistics derived from experimental data to help firms assess the potential benefit of applying the method.

Experimental results

Research questions

• RQ1 Can principal stratification reduce the variance of ATE estimates in advertising experiments with noisy responses?
• RQ2 How effective are pre-randomization covariates—like past purchase behavior—in predicting stratum membership in advertising experiments?
• RQ3 Does the inclusion of covariates that predict stratum membership improve the precision of ATE estimation?
• RQ4 Under what conditions is the principal stratification method expected to be beneficial in practice?

Key findings

• The principal stratification model reduced the variance of ATE estimates by 36–57% across five catalog mailing experiments with approximately 140,000 participants.
• Customers who had not purchased recently were significantly more likely to belong to the 'never buy' stratum.
• Past response to similar advertising, even when biased due to targeted past campaigns, was a strong predictor of stratum membership.
• The method effectively isolates the large group of always buyers—whose treatment effects are exactly zero—thereby reducing overall variance in ATE estimation.
• Firms can use simple summary statistics from their own data to determine whether applying the method is likely to yield meaningful variance reduction.