[Paper Review] Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability

Jonathan Crabbé, Alicia Curth+2 more|arXiv (Cornell University)|Jun 16, 2022

Advanced Causal Inference Techniques6 citations

TL;DR

This paper introduces the ITErpretability benchmark to evaluate heterogeneous treatment effect (HTE) models through post-hoc feature importance, revealing that model interpretability—particularly in identifying predictive covariates—varies significantly across confounding types. It finds that predictive confounding most severely degrades attribution performance, while balancing regularizers in models like CFRNet can reduce interpretability despite improving estimation error.

ABSTRACT

Estimating personalized effects of treatments is a complex, yet pervasive problem. To tackle it, recent developments in the machine learning (ML) literature on heterogeneous treatment effect estimation gave rise to many sophisticated, but opaque, tools: due to their flexibility, modularity and ability to learn constrained representations, neural networks in particular have become central to this literature. Unfortunately, the assets of such black boxes come at a cost: models typically involve countless nontrivial operations, making it difficult to understand what they have learned. Yet, understanding these models can be crucial -- in a medical context, for example, discovered knowledge on treatment effect heterogeneity could inform treatment prescription in clinical practice. In this work, we therefore use post-hoc feature importance methods to identify features that influence the model's predictions. This allows us to evaluate treatment effect estimators along a new and important dimension that has been overlooked in previous work: We construct a benchmarking environment to empirically investigate the ability of personalized treatment effect models to identify predictive covariates -- covariates that determine differential responses to treatment. Our benchmarking environment then enables us to provide new insight into the strengths and weaknesses of different types of treatment effects models as we modulate different challenges specific to treatment effect estimation -- e.g. the ratio of prognostic to predictive information, the possible nonlinearity of potential outcomes and the presence and type of confounding.

Motivation & Objective

• To address the lack of interpretability evaluation in HTE model benchmarking, which traditionally focuses only on PEHE metrics.
• To investigate how different types of confounding—prognostic, predictive, and non-confounded—affect the ability of HTE models to identify true predictive covariates.
• To develop a transparent, semi-synthetic benchmarking environment that enables systematic evaluation of feature importance in black-box HTE estimators.
• To assess whether existing regularization techniques, such as balancing in CFRNet, improve or hinder interpretability despite improving estimation accuracy.
• To provide new insights into the trade-offs between estimation precision and model interpretability in personalized treatment effect estimation.

Proposed method

• Construct a semi-synthetic dataset using a transparent data-generating process with known predictive covariates (ground truth).
• Train black-box HTE estimators (e.g., neural networks, CFRNet) on the semi-synthetic data to estimate conditional average treatment effects (CATE).
• Apply post-hoc feature importance methods (e.g., LIME, SHAP) to interpret the predictions of each trained HTE model.
• Define a new metric, Attr_pred, to measure the overlap between top-ranked feature importance scores and the true predictive covariates.
• Systematically vary confounding types (prognostic, predictive, non-confounded) and propensity score scales to evaluate robustness of interpretability.
• Compare performance across models using both Attr_pred (interpretability) and PEHE (estimation accuracy) to reveal trade-offs.

Experimental results

Research questions

• RQ1 How does the presence and type of confounding (prognostic, predictive, non-confounded) affect the ability of HTE models to identify true predictive covariates?
• RQ2 To what extent do balancing regularizations in models like CFRNet improve or impair interpretability, despite enhancing estimation accuracy?
• RQ3 How do attribution scores (feature importance) correlate with estimation error (PEHE) across different confounding settings?
• RQ4 Does the transparency of the data-generating process enable reliable benchmarking of interpretability in black-box HTE models?
• RQ5 Are there systematic differences in interpretability performance between models trained on datasets with varying degrees of treatment assignment bias?

Key findings

• Attribution performance, measured by Attr_pred, systematically deteriorates as treatment assignment bias increases, particularly under predictive confounding.
• Performance degradation is most severe under predictive confounding, where reduced covariate variation within treatment groups hinders model learning of true effect modifiers.
• Surprisingly, Attr_pred performance is least degraded under prognostic confounding, as covariate imbalance partially offsets misattribution to non-predictive features.
• CFRNet’s balancing regularization causes a significant drop in Attr_pred under predictive confounding, indicating a trade-off between estimation accuracy and interpretability.
• Despite improved PEHE in prognostic confounding, Attr_pred performance remains relatively stable, suggesting balancing regularization may reduce estimation accuracy of prognostic components.
• The benchmark reveals that PEHE alone is insufficient for evaluating HTE models, as models with similar PEHE can differ drastically in interpretability and feature discovery capability.