[Paper Review] Sensitivity Analysis of Individual Treatment Effects: A Robust Conformal Inference Approach

Ying Jin, Zhimei Ren+1 more|arXiv (Cornell University)|Nov 23, 2021

Advanced Causal Inference Techniques42 references4 citations

TL;DR

This paper proposes a model-free, conformal inference-based framework for sensitivity analysis of individual treatment effects (ITEs) under unmeasured confounding. It introduces the \Gamma-value—a measure of the minimum confounding strength needed to explain away a positive ITE—ensuring valid, marginal coverage for counterfactual predictions even when confounding is present, with theoretical guarantees under the marginal sensitivity model.

ABSTRACT

We propose a model-free framework for sensitivity analysis of individual treatment effects (ITEs), building upon ideas from conformal inference. For any unit, our procedure reports the $Γ$-value, a number which quantifies the minimum strength of confounding needed to explain away the evidence for ITE. Our approach rests on the reliable predictive inference of counterfactuals and ITEs in situations where the training data is confounded. Under the marginal sensitivity model of Tan (2006), we characterize the shift between the distribution of the observations and that of the counterfactuals. We first develop a general method for predictive inference of test samples from a shifted distribution; we then leverage this to construct covariate-dependent prediction sets for counterfactuals. No matter the value of the shift, these prediction sets (resp. approximately) achieve marginal coverage if the propensity score is known exactly (resp. estimated). We describe a distinct procedure also attaining coverage, however, conditional on the training data. In the latter case, we prove a sharpness result showing that for certain classes of prediction problems, the prediction intervals cannot possibly be tightened. We verify the validity and performance of the new methods via simulation studies and apply them to analyze real datasets.

Motivation & Objective

• To develop a model-free framework for sensitivity analysis of individual treatment effects (ITEs) in the presence of unmeasured confounding.
• To quantify the robustness of ITE inferences by identifying the minimum confounding strength required to negate a positive ITE prediction.
• To ensure valid marginal coverage of prediction intervals for counterfactuals and ITEs under distributional shifts due to confounding.
• To provide a procedure that maintains valid inference even when the propensity score is estimated, not known exactly.
• To offer a sharpness result showing that prediction intervals cannot be tightened beyond a certain bound under certain conditions.

Proposed method

• Uses the marginal sensitivity model (Tan, 2006) to characterize the shift between observed and counterfactual distributions under unmeasured confounding.
• Develops a general method for predictive inference under distributional shift, enabling reliable counterfactual prediction for test samples.
• Constructs covariate-dependent prediction sets for counterfactuals and ITEs that achieve approximate marginal coverage when the propensity score is estimated.
• Employs conformal inference to generate prediction intervals that are valid under the assumed shift model, with theoretical guarantees on coverage.
• Introduces a distinct procedure that achieves conditional coverage given the training data, proving a sharpness result for prediction interval optimality.
• Leverages worst-case distributional shifts to compute sharp bounds on counterfactual expectations, enabling robust inference under uncertainty.

Experimental results

Research questions

• RQ1 How can we assess the robustness of individual treatment effect estimates to unmeasured confounding in observational studies?
• RQ2 What is the minimum strength of unmeasured confounding required to explain away a positive ITE prediction for a given unit?
• RQ3 Can we construct valid prediction intervals for counterfactuals and ITEs when the training data is confounded and the ignorability assumption fails?
• RQ4 How does the proposed method maintain coverage under estimated propensity scores, and what are the theoretical limits on prediction interval sharpness?
• RQ5 Can the framework be extended to simultaneous inference on multiple ITEs with proper error control?

Key findings

• For a real dataset on mindset interventions, 20.46% of treated test samples had \Gamma-values greater than 1, indicating robust evidence of positive ITEs under mild confounding.
• At \Gamma=2, only 6.80% of treated test samples had \Gamma-values exceeding 2, showing that strong evidence for positive ITEs persists under moderate confounding.
• With Algorithm 1, 2% of test samples had \Gamma-values greater than 5, and with Algorithm 2, 2.5% had \Gamma-values exceeding 10, indicating strong robustness for a subset of individuals.
• The method guarantees that the probability of incorrectly classifying a negative ITE as positive is bounded by \alpha=0.1 under confounding strength \Gamma.
• Empirical evaluation showed that only 3.58% of treated test samples were incorrectly classified as having positive ITEs at \Gamma=1, with low error rates increasing only slightly at higher \Gamma levels.
• Algorithm 2 produced slightly stronger evidence against confounding but was less stable than Algorithm 1, with both methods maintaining valid coverage under the theoretical guarantees.