[Paper Review] A unified survey of treatment effect heterogeneity modeling and uplift modeling

Weijia Zhang, Jiuyong Li+1 more|arXiv (Cornell University)|Jul 14, 2020

Advanced Causal Inference Techniques103 references4 citations

TL;DR

This paper presents a unified framework for treatment effect heterogeneity modeling and uplift modeling under the potential outcomes causal inference framework, demonstrating their shared goal of estimating conditional average treatment effects (CATE). It systematically compares methods from both communities using consistent notation, evaluates performance across datasets, and provides guidelines for method and software selection with emphasis on interpretability, accuracy, and data-specific performance trade-offs.

ABSTRACT

A central question in many fields of scientific research is to determine how an outcome would be affected by an action, or to measure the effect of an action (a.k.a treatment effect). In recent years, a need for estimating the heterogeneous treatment effects conditioning on the different characteristics of individuals has emerged from research fields such as personalized healthcare, social science, and online marketing. To meet the need, researchers and practitioners from different communities have developed algorithms by taking the treatment effect heterogeneity modeling approach and the uplift modeling approach, respectively. In this paper, we provide a unified survey of these two seemingly disconnected yet closely related approaches under the potential outcome framework. We then provide a structured survey of existing methods by emphasizing on their inherent connections with a set of unified notations to make comparisons of the different methods easy. We then review the main applications of the surveyed methods in personalized marketing, personalized medicine, and social studies. Finally, we summarize the existing software packages and present discussions based on the use of methods on synthetic, semi-synthetic and real world data sets and provide some general guidelines for choosing methods.

Motivation & Objective

• To bridge the gap between two isolated research communities—treatment effect heterogeneity modeling and uplift modeling—by showing their shared objective under the potential outcomes framework.
• To provide a structured, comparative survey of existing methods from both communities using unified notation to highlight inherent connections and differences.
• To evaluate the performance of various CATE estimation methods across synthetic, semi-synthetic, and real-world datasets, identifying strengths and limitations.
• To summarize available open-source software packages and tools for CATE estimation, assessing their coverage and applicability to different data types.
• To offer practical guidelines for selecting appropriate methods and tools based on data characteristics, interpretability needs, and performance trade-offs.

Proposed method

• Uses the potential outcomes framework with the overlap, SUTVA, and unconfoundedness assumptions to formally unify treatment effect heterogeneity modeling and uplift modeling.
• Defines the conditional average treatment effect (CATE) as the core target, estimating E[Y|X=x, T=1] - E[Y|X=x, T=0] where X is covariates, T is treatment, and Y is outcome.
• Classifies methods into single-model, two-model, and X-Learner approaches, with the X-Learner using separate models for treatment and control groups and a meta-learner for CATE estimation.
• Employs ensemble methods (e.g., X-Learner with BART) to improve stability and performance, especially on unbalanced datasets.
• Evaluates methods using standard benchmarks like IHDP and Hillstrom’s Email Advertisement datasets, measuring accuracy via RMSE and other CATE-specific metrics.
• Compares deep learning-based methods with tree-based and model-based approaches, highlighting sensitivity to hyperparameter tuning and lack of ground-truth CATE values.

Experimental results

Research questions

• RQ1 How are treatment effect heterogeneity modeling and uplift modeling formally connected under the potential outcomes framework?
• RQ2 What are the key methodological differences and similarities between the two communities’ approaches to CATE estimation?
• RQ3 How do different CATE estimation methods (e.g., single-model, two-model, X-Learner, ensemble, deep learning) perform across diverse datasets in terms of accuracy and stability?
• RQ4 What are the trade-offs between interpretability, training efficiency, and predictive accuracy in CATE estimation methods?
• RQ5 Which open-source software tools best support the implementation and evaluation of CATE estimation methods for real-world applications?

Key findings

• The X-Learner and two-model approaches show strong performance, particularly on unbalanced data, and are flexible in combining existing supervised learning models.
• Ensemble methods exhibit stable performance across datasets but suffer from high computational cost and poor interpretability.
• Single-model approaches generally yield lower accuracy, especially when treatment and control groups are imbalanced.
• Deep learning-based methods require extensive hyperparameter tuning and are sensitive to data variability, with performance highly dependent on dataset-specific optimization.
• No single method consistently outperforms others across all datasets, indicating that method selection is highly data-dependent.
• Tree-based models offer better interpretability with minimal tuning, while black-box models like deep learning achieve higher accuracy only with careful, dataset-specific configuration.