[Paper Review] Personalised Decision-Making without Counterfactuals

A. P. Dawid, Stephen Senn|arXiv (Cornell University)|Jan 27, 2023

Health Systems, Economic Evaluations, Quality of LifeEconomics, Econometrics and Finance3 citations

TL;DR

This paper challenges Judea Pearl's counterfactual-based approach to personalized decision-making in healthcare, arguing it is theoretically flawed and practically dangerous. It advocates for a traditional decision-theoretic framework using observed conditional probabilities (Pr(Y=1|L=l,X←x)) to maximize recovery rates, showing that counterfactual reasoning introduces unwarranted assumptions and can lead to harmful clinical decisions when applied in practice.

ABSTRACT

This article is a response to recent proposals by Pearl and others for a new approach to personalised treatment decisions, in contrast to the traditional one based on statistical decision theory. We argue that this approach is dangerously misguided and should not be used in practice.

Motivation & Objective

• To challenge the validity and safety of using counterfactual reasoning in personalized treatment decisions as proposed by Pearl and colleagues.
• To demonstrate that traditional decision-theoretic methods based on observed conditional probabilities are superior and more reliable for clinical decision-making.
• To highlight the critical assumptions underlying counterfactual models that are unlikely to hold in real-world medical settings.
• To show that counterfactual-based approaches can lead to suboptimal or harmful treatment decisions due to unverifiable causal assumptions.
• To advocate for the use of interventionist utility and observed data distributions as a safer, more practical alternative to counterfactual utility frameworks.

Proposed method

• Uses a decision-theoretic framework based on observed conditional probabilities Pr(Y=1|L=l,X←x) to determine optimal treatment for a patient with covariate profile L=l.
• Defines the conditional average treatment effect (CATE) as CATE(l) = Pr(Y=1|L=l,X←1) - Pr(Y=1|L=l,X←0), recommending treatment when CATE(l) > 0.
• Applies the interventionist utility approach, which relies on actual interventional outcomes rather than unobserved counterfactuals.
• Introduces bounds on probabilities of benefit (PB) and harm (PH) using experimental data and, when available, observational data with covariates like 'intention to treat' (X*).
• Uses the parameters τ (ATE) and ρ to express interventional probabilities in matrix form P(τ, ρ), enabling bounds on PB and PH under uncertainty.
• Combines experimental and observational data to identify the distribution of intention-to-treat covariates (X*), improving bounds on PB and PH even when L is unobserved.

Experimental results

Research questions

• RQ1 Is the counterfactual-based approach to personalized decision-making, as proposed by Pearl et al., theoretically sound and practically safe?
• RQ2 Can traditional decision-theoretic methods based on observed conditional probabilities outperform counterfactual models in treatment selection?
• RQ3 What are the critical assumptions underlying counterfactual models that are unlikely to hold in real-world clinical data?
• RQ4 How can observational data on intention-to-treat behavior be used to improve bounds on probabilities of benefit and harm?
• RQ5 Under what conditions do bounds on PB and PH shrink to point estimates, and how does this affect treatment decisions?

Key findings

• The counterfactual-based approach to personalized medicine is dangerously misguided and risks causing real clinical harm due to unverifiable assumptions.
• In the absence of covariate information (L), treatment decisions should be based on unconditional interventional probabilities, not on imputed or counterfactual values.
• When L is unobserved, the optimal strategy is to treat if the conditional average treatment effect (CATE) is positive, as demonstrated in Example 1 with CATE(l) = 0.28.
• Using observational data on intention-to-treat (X*) in combination with experimental data allows for tighter bounds on the probability of benefit (PB) and harm (PH), even when L is unobserved.
• The bounds on PB and PH shrink to point estimates when the joint distribution of (X, Y, X*) is fully identifiable from combined experimental and observational data.
• The interventionist utility approach—based on actual interventional outcomes—yields the same optimal decisions as the counterfactual approach only when counterfactual assumptions are valid; otherwise, it avoids spurious conclusions.