[Paper Review] True to the Model or True to the Data?

Hugh Chen, Joseph D. Janizek+2 more|arXiv (Cornell University)|Jun 29, 2020

Explainable Artificial Intelligence (XAI)Computer Science19 references37 citations

TL;DR

The paper compares observational and interventional Shapley values for feature attribution, showing application-dependent benefits, and introduces an efficient linear-model approach with real-data examples (credit risk and biology).

ABSTRACT

A variety of recent papers discuss the application of Shapley values, a concept for explaining coalitional games, for feature attribution in machine learning. However, the correct way to connect a machine learning model to a coalitional game has been a source of controversy. The two main approaches that have been proposed differ in the way that they condition on known features, using either (1) an interventional or (2) an observational conditional expectation. While previous work has argued that one of the two approaches is preferable in general, we argue that the choice is application dependent. Furthermore, we argue that the choice comes down to whether it is desirable to be true to the model or true to the data. We use linear models to investigate this choice. After deriving an efficient method for calculating observational conditional expectation Shapley values for linear models, we investigate how correlation in simulated data impacts the convergence of observational conditional expectation Shapley values. Finally, we present two real data examples that we consider to be representative of possible use cases for feature attribution -- (1) credit risk modeling and (2) biological discovery. We show how a different choice of value function performs better in each scenario, and how possible attributions are impacted by modeling choices.

Motivation & Objective

• Motivate the debate between observational and interventional conditioning for Shapley values in ML explanations.
• Develop a linear-model framework to compute observational Shapley values efficiently.
• Explore how feature correlation affects convergence and attribution in observational SHAP.
• Demonstrate when being true to the model versus true to the data is preferable using real datasets (credit risk and biology).
• Discuss practical implications and limitations for applying Shapley-based explanations.

Proposed method

• Define v(S) using observational conditional expectation and interventional conditional expectation.
• Derive analytic form for Shapley values in linear models with f(x)=βx+b.
• Assume x ~ N(μ,Σ) and project conditional expectations to the full feature space via appropriate projections.
• Provide a fast computation strategy by factoring expressions into T(μ) and T(x) to reuse exponential terms across samples.
• Discuss Monte Carlo sampling as an approximation and compare computational costs (exact vs. approximate).
• Apply the approaches to real datasets (LendingClub loan defaults and NHANES mortality) and an RNA-seq-based biological discovery scenario.

Experimental results

Research questions

• RQ1 Which conditioning (observational vs interventional) should be used for Shapley attributions in different applications?
• RQ2 How does feature correlation affect convergence and attribution in observational SHAP for linear models?
• RQ3 Can modeling choices (e.g., Elastic Net) mitigate the data-true attribution issue for correlated features?
• RQ4 Do observational and interventional SHAP values align with being true to the data vs true to the model in practical use cases?
• RQ5 What are the computational trade-offs between exact observational SHAP calculations and model-based alternatives in real-world tasks?

Key findings

• Observational SHAP values converge more slowly and spread credit among correlated features, especially as feature correlation increases.
• Interventional SHAP values tend to be more true to the model, assigning credit to features explicitly used by the model and yielding larger impact when perturbing those features.
• Observational SHAP can attribute importance to features not used by the model due to correlations (e.g., BMI influencing attribution when correlated with other features).
• In a credit-risk scenario, interventions based on interventional SHAP explanations decrease predicted default risk more effectively than observational explanations.
• In a biological discovery scenario, observational SHAP better recovers true causal features when using a Lasso model, while Elastic Net modeling (which spreads credit among correlated features) can align explanations with data structure; nonetheless, Elastic Net with interventional SHAP often offers a computationally faster route.
• Elastic Net models can spread credit among correlated features, mitigating the need for computationally heavy observational SHAP calculations.