[Paper Review] Heterogeneous Treatment Effects in Digital Experimentation

Matt Taddy, Matt Gardner+2 more|arXiv (Cornell University)|Dec 30, 2014

Advanced Causal Inference Techniques21 references8 citations

TL;DR

This paper introduces a fast, scalable Bayesian nonparametric method for analyzing heterogeneous treatment effects in digital experiments, using linear projections and regression trees (CART/Random Forests) with uncertainty quantification. It finds that post-stratification via linear regression offers minimal variance reduction in large-scale digital experiments, while ensembles of trees (forests) provide more reliable inference than single trees.

ABSTRACT

Randomized controlled trials play an important role in how Internet companies predict the impact of policy decisions and product changes. In these `digital experiments', different units (people, devices, products) respond differently to the treatment. This article presents a fast and scalable Bayesian nonparametric analysis of such heterogeneous treatment effects and their measurement in relation to observable covariates. New results and algorithms are provided for quantifying the uncertainty associated with treatment effect measurement via both linear projections and nonlinear regression trees (CART and Random Forests). For linear projections, our inference strategy leads to results that are mostly in agreement with those from the frequentist literature. We find that linear regression adjustment of treatment effect averages (i.e., post-stratification) can provide some variance reduction, but that this reduction will be vanishingly small in the low-signal and large-sample setting of digital experiments. For regression trees, we provide uncertainty quantification for the machine learning algorithms that are commonly applied in tree-fitting. We argue that practitioners should look to ensembles of trees (forests) rather than individual trees in their analysis. The ideas are applied on and illustrated through an example experiment involving 21 million unique users of this http URL.

Motivation & Objective

• To address the challenge of estimating heterogeneous treatment effects in large-scale digital experiments where units respond differently to interventions.
• To develop a computationally efficient and scalable Bayesian nonparametric framework for uncertainty quantification in treatment effect estimation.
• To evaluate the effectiveness of linear regression adjustment (post-stratification) in reducing variance under low-signal, high-sample settings typical of digital experiments.
• To provide uncertainty quantification for machine learning models like CART and Random Forests in the context of treatment effect analysis.
• To advocate for the use of tree ensembles (forests) over single trees in digital experimentation due to improved reliability and robustness.

Proposed method

• Uses Bayesian nonparametric inference to model heterogeneous treatment effects, enabling flexible modeling without strong parametric assumptions.
• Applies linear projections with hierarchical priors to estimate average treatment effects adjusted for observable covariates.
• Employs a posterior predictive distribution to quantify uncertainty in treatment effect estimates under linear models.
• Adapts Bayesian inference to regression trees (CART) by placing priors on split points and node parameters, enabling uncertainty quantification.
• Extends the framework to Random Forests by aggregating posterior distributions across multiple trees to improve stability and reduce overfitting.
• Employs scalable computation techniques to handle large datasets, such as the 21 million user experiment studied.

Experimental results

Research questions

• RQ1 How effective is linear regression adjustment (post-stratification) at reducing variance in treatment effect estimates within large-scale digital experiments?
• RQ2 What is the appropriate way to quantify uncertainty in treatment effect estimates when using tree-based models like CART and Random Forests?
• RQ3 How do the results from Bayesian nonparametric methods compare to frequentist approaches in linear treatment effect models?
• RQ4 In what settings do individual trees outperform or underperform relative to tree ensembles in treatment effect estimation?
• RQ5 What are the practical implications of using Bayesian uncertainty quantification in real-world digital experiments with massive sample sizes?

Key findings

• Post-stratification via linear regression adjustment provides only minimal variance reduction in low-signal, large-sample digital experiments, rendering its benefit negligible in practice.
• The Bayesian nonparametric approach produces results largely consistent with frequentist methods for linear models, validating its reliability.
• Uncertainty quantification for tree-based models is feasible and provides meaningful posterior intervals, improving interpretability over point estimates.
• Ensembles of trees (Random Forests) yield more stable and reliable treatment effect estimates than individual trees, supporting their use in practice.
• The proposed method scales efficiently to datasets with tens of millions of units, as demonstrated in a 21 million user experiment.
• The framework enables practitioners to quantify uncertainty in treatment effects across diverse subpopulations defined by covariates, enhancing decision-making in digital product development.