[Paper Review] A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution

Aria Khademi, Vasant Honavar|arXiv (Cornell University)|Aug 1, 2020

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper proposes a causal attribution framework to interpret black box predictive models by estimating the causal effect of each input feature on model outputs using the Rubin-Neyman potential outcomes framework. It demonstrates that causal attribution provides more reliable and trustworthy explanations than correlation-based methods, especially in the presence of confounders, and validates the approach on synthetic and real-world datasets including digit classification and Parkinson’s disease prediction.

ABSTRACT

With the increasing adoption of predictive models trained using machine learning across a wide range of high-stakes applications, e.g., health care, security, criminal justice, finance, and education, there is a growing need for effective techniques for explaining such models and their predictions. We aim to address this problem in settings where the predictive model is a black box; That is, we can only observe the response of the model to various inputs, but have no knowledge about the internal structure of the predictive model, its parameters, the objective function, and the algorithm used to optimize the model. We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output, from observations of the model inputs and the corresponding outputs. We estimate the causal effects of model inputs on model output using variants of the Rubin Neyman potential outcomes framework for estimating causal effects from observational data. We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions. We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction. Because our approach does not require knowledge about the predictive model algorithm and is free of assumptions regarding the black box predictive model except that its input-output responses be observable, it can be applied, in principle, to any black box predictive model.

Motivation & Objective

• To address the critical need for trustworthy, causal explanations of black box predictive models in high-stakes domains such as healthcare, finance, and criminal justice.
• To overcome the limitations of correlation-based interpretation methods that fail to account for confounders and produce unreliable attributions.
• To develop a model-agnostic method for interpreting black box models using only observable input-output behavior, without requiring knowledge of the model’s internal structure.
• To ground model interpretation in causality, ensuring explanations answer 'why' questions in a way that supports trust and accountability.
• To evaluate the effectiveness of causal attribution in real-world and synthetic settings, demonstrating its superiority over traditional feature importance and gradient-based methods.

Proposed method

• The method estimates causal effects of model inputs on outputs using the potential outcomes framework, treating each input as a 'treatment' and the model output as the 'outcome'.
• It applies observational data from the black box model’s input-output responses to estimate average causal effects via methods such as inverse probability weighting and propensity score matching.
• The approach assumes strong ignorability, requiring that all confounders are observed and that there are no unmeasured confounders affecting both inputs and outputs.
• Causal effects are estimated using multiple techniques to assess consistency; reliable attributions are prioritized when estimates agree across methods.
• The framework is model-agnostic and does not require access to model parameters, architecture, or training procedure.
• It leverages structural causal models or background knowledge to identify and control for confounders when estimating causal effects from observational data.

Experimental results

Research questions

• RQ1 Can causal attribution provide more reliable and trustworthy explanations of black box predictive models than correlation-based methods?
• RQ2 How can causal effects of individual features on model outputs be estimated from only input-output observations without access to model internals?
• RQ3 In the presence of confounders, does causal attribution outperform traditional feature importance or gradient-based explanation methods?
• RQ4 How consistent are causal attributions across different estimation techniques, and can agreement among methods improve confidence in explanations?
• RQ5 Can the proposed causal attribution framework be effectively applied to real-world and synthetic datasets, including deep neural networks?

Key findings

• The causal attribution framework successfully identifies the true input features responsible for model outputs in a synthetic dataset where ground-truth causal relationships are known.
• On the handwritten digit classification dataset, the method provides consistent and interpretable attributions that align with known salient features in the images.
• For Parkinson’s disease severity prediction, the approach identifies clinically relevant features such as voice and motor function metrics, demonstrating relevance to real-world medical applications.
• The method outperforms correlation-based approaches in the presence of confounders, as evidenced by more accurate and stable attributions across multiple estimation techniques.
• Disagreement in causal effect estimates across methods signals potential violations of strong ignorability, highlighting cases where explanations should be treated with caution.
• The framework is applicable to any black box model as long as input-output behavior is observable, making it broadly generalizable across machine learning applications.