[Paper Review] Bias Correction For Paid Search In Media Mix Modeling

Aiyou Chen, David Chan+5 more|arXiv (Cornell University)|Jul 9, 2018

Consumer Market Behavior and Pricing20 references3 citations

TL;DR

This paper proposes a statistically principled bias correction method for paid search in media mix modeling (MMM) using the back-door criterion from causal inference to address selection bias from ad targeting. By adjusting for confounders like consumer demand and organic search, the method yields approximately unbiased estimates of search ad ROAS, validated through case studies against randomized experiments and real-world data.

ABSTRACT

Evaluating the return on ad spend (ROAS), the causal effect of advertising on sales, is critical to advertisers for understanding the performance of their existing marketing strategy as well as how to improve and optimize it. Media Mix Modeling (MMM) has been used as a convenient analytical tool to address the problem using observational data. However it is well recognized that MMM suffers from various fundamental challenges: data collection, model specification and selection bias due to ad targeting, among others \citep{chan2017,wolfe2016}. In this paper, we study the challenge associated with measuring the impact of search ads in MMM, namely the selection bias due to ad targeting. Using causal diagrams of the search ad environment, we derive a statistically principled method for bias correction based on the extit{back-door} criterion \citep{pearl2013causality}. We use case studies to show that the method provides promising results by comparison with results from randomized experiments. We also report a more complex case study where the advertiser had spent on more than a dozen media channels but results from a randomized experiment are not available. Both our theory and empirical studies suggest that in some common, practical scenarios, one may be able to obtain an approximately unbiased estimate of search ad ROAS.

Motivation & Objective

• To address selection bias in media mix modeling (MMM) caused by ad targeting in paid search campaigns.
• To develop a statistically principled method for correcting bias in ROAS estimation using causal inference.
• To validate the method using both randomized experiments and real-world case studies with multi-channel advertising data.
• To demonstrate that unbiased ROAS estimation is achievable in practical, common scenarios despite challenges like funnel effects and confounding factors.
• To provide a framework applicable to real-world MMM systems without requiring full experimental validation.

Proposed method

• Uses causal diagrams to model the search ad environment and identify confounders such as consumer demand and organic search.
• Applies the back-door criterion from do-calculus to identify a set of observed confounders that, when conditioned on, block back-door paths and enable unbiased causal estimation.
• Derives a bias correction method (SBC) that adjusts for confounding variables like search query volume and organic clicks to isolate the true causal effect of paid search spend on sales.
• Implements the method using regression models where the outcome is sales and predictors include paid search spend, adjusted for identified confounders.
• Validates the method by comparing corrected estimates against results from randomized experiments as a gold standard.
• Handles complex scenarios by excluding organic clicks as a confounder when they are influenced by paid clicks, avoiding feedback loop bias.

Experimental results

Research questions

• RQ1 Can selection bias in paid search MMM be corrected using a principled causal inference approach?
• RQ2 Does the proposed back-door criterion-based method yield approximately unbiased ROAS estimates in real-world settings?
• RQ3 How does the bias-corrected estimate compare to results from randomized experiments in controlled case studies?
• RQ4 In what practical scenarios does the method remain valid despite confounding factors like organic search and demand fluctuations?
• RQ5 Can the method be applied effectively in multi-channel advertising environments where randomized experiments are unavailable?

Key findings

• The proposed bias correction method (SBC) significantly reduces over-estimation of ROAS caused by selection bias from ad targeting.
• In case studies with randomized experiments as ground truth, SBC estimates were consistently closer to the experimental ROAS than naive regression models.
• The method achieved approximately unbiased ROAS estimation in common, practical scenarios, even when full experimental validation was not available.
• Controlling for organic clicks as a confounder can introduce bias if the direction of causality is misidentified; the study recommends excluding it when influenced by paid clicks.
• The method remains robust under various confounding conditions, including external events like data breaches or competitor pricing changes, when properly identified and adjusted for.
• The approach provides a viable alternative to randomized experiments for measuring paid search effectiveness in real-world MMM applications.