[Paper Review] Feature relevance quantification in explainable AI: A causal problem

Dominik Janzing, Lenon Minorics+1 more|arXiv (Cornell University)|Oct 29, 2019

Explainable Artificial Intelligence (XAI)Computer Science21 references155 citations

TL;DR

The paper argues that feature relevance should be quantified using interventional (do-operator) marginals rather than observational conditionals, and analyzes how Shapley values behave under these choices, critiquing SHAP's use of conditional expectations as approximations.

ABSTRACT

We discuss promising recent contributions on quantifying feature relevance using Shapley values, where we observed some confusion on which probability distribution is the right one for dropped features. We argue that the confusion is based on not carefully distinguishing between observational and interventional conditional probabilities and try a clarification based on Pearl's seminal work on causality. We conclude that unconditional rather than conditional expectations provide the right notion of dropping features in contradiction to the theoretical justification of the software package SHAP. Parts of SHAP are unaffected because unconditional expectations (which we argue to be conceptually right) are used as approximation for the conditional ones, which encouraged others to `improve' SHAP in a way that we believe to be flawed.

Motivation & Objective

• Clarify which probability distribution is appropriate for dropping features when attributing model output to inputs.
• Distinguish observational vs. interventional conditional probabilities in feature attribution.
• Assess how Shapley values behave when using marginal vs. conditional expectations for attribution relative to the overall mean.
• Evaluate whether using interventional expectations resolves issues like non-sensitivity and symmetry in attribution.
• Provide guidance on how the SHAP framework relates to causal perspectives and where it remains valid.

Proposed method

• Review axiomatic foundations of feature attribution with a fixed baseline, including integrated gradients and Shapley values.
• Define two candidate simplified functions f_T: (i) conditional expectation given X_T=x_T and (ii) marginal expectation, and discuss their implications for attribution.
• Use Pearl’s do-calculus to motivate that interventional (do) expectations correspond to marginal expectations when attributing to X_T.
• Present a causal diagram and explain how interventions on X_T break certain dependencies, justifying marginal expectations as the correct notion of dropping features.
• Prove that using conditional expectations can yield non-sensical attributions (e.g., non-zero attributions for irrelevant features).
• Describe how KernelSHAP computes Shapley values via a weighted least squares formulation and how g(T) is approximated in practice (marginal vs conditional).

Experimental results

Research questions

• RQ1 What is the correct probabilistic notion of dropping features when attributing model output to inputs?
• RQ2 Do observational conditionals or interventional (do) probabilities better capture feature relevance in attribution?
• RQ3 How do Shapley values behave when attribution is defined relative to the marginal expectation versus the conditional expectation?
• RQ4 Can interventional reasoning resolve issues like false sensitivities and symmetry violations in attribution frameworks?
• RQ5 How does SHAP relate to causal perspectives, and where is it conceptually sound or flawed?

Key findings

• Marginal (interventional) expectations align with do-operations and are conceptually correct for attributing differences to features.
• Using conditional expectations can produce non-zero attributions for irrelevant features, violating intuitive sensitivity.
• SHAP’s kernel-based approximation remains largely unaffected where it approximates marginal expectations, but approaches that rely on conditional expectations are conceptually flawed.
• Empirical simulations show marginal-based Shapley values better match true contributions in Gaussian and real data settings.
• The paper clarifies that some SHAP components are acceptable approximations, while others require revision to reflect causal interventions.
• Acausal or purely observational attributions risk misrepresenting feature influence due to confounding and dependent feature distributions.