[Paper Review] An unexpected unity among methods for interpreting model predictions

Scott Lundberg, Su‐In Lee|arXiv (Cornell University)|Nov 22, 2016

Explainable Artificial Intelligence (XAI)5 references104 citations

TL;DR

The paper shows that a model-agnostic additive representation of feature importance, based on Expectation Shapley (ES) values, unifies and justifies a range of model interpretation methods, and links them to LIME, DeepLIFT, and Layer-wise Relevance Propagation as ES value approximations.

ABSTRACT

Understanding why a model made a certain prediction is crucial in many data science fields. Interpretable predictions engender appropriate trust and provide insight into how the model may be improved. However, with large modern datasets the best accuracy is often achieved by complex models even experts struggle to interpret, which creates a tension between accuracy and interpretability. Recently, several methods have been proposed for interpreting predictions from complex models by estimating the importance of input features. Here, we present how a model-agnostic additive representation of the importance of input features unifies current methods. This representation is optimal, in the sense that it is the only set of additive values that satisfies important properties. We show how we can leverage these properties to create novel visual explanations of model predictions. The thread of unity that this representation weaves through the literature indicates that there are common principles to be learned about the interpretation of model predictions that apply in many scenarios.

Motivation & Objective

• Motivate the need for interpreting predictions from complex models to foster trust and insight.
• Propose and formalize an additive feature-importance representation that is optimal under certain axioms.
• Show how ES values unify existing explanation methods and justify their design choices.
• Provide visualization strategies for interpreting and diagnosing model predictions across datasets.

Proposed method

• Adopts a prediction explanation framework based on Shapley values to distribute credit among input features.
• Derives ES values as the unique additive solution satisfying efficiency, symmetry, and monotonicity axioms.
• Defines a specific kernel (Shapley kernel) and loss/regularization setup that yields ES values as the solution.
• Connects ES values to existing methods (LIME, DeepLIFT, Layer-wise Relevance Propagation) as approximations or special cases.
• Proposes a visualization approach that represents ES values as a force-like additive decomposition of the prediction.
• Discusses sample efficiency and kernel importance for estimating ES values and comparing with Shapley regression and LIME.

Experimental results

Research questions

• RQ1 Can a single additive, game-theoretic framework capture the behavior of various prediction explanation methods?
• RQ2 What axioms constrain feature attribution methods, and what is the unique solution under these constraints?
• RQ3 How do existing methods like LIME, DeepLIFT, and LRP relate to ES values when interpreting model predictions?
• RQ4 What are the practical implications and visualizations that arise from interpreting ES values in real models?
• RQ5 How does kernel choice affect the estimation and interpretation of feature attributions?

Key findings

• ES values provide an optimal, unique additive decomposition of a model’s prediction under efficiency, symmetry, and monotonicity axioms.
• The Shapley kernel and corresponding loss/regularization yield ES values as the interpretation solution; the kernel enforces f(x)=sum of attributions.
• ES values unify and rationalize several interpretation methods, showing they are approximations or special cases of ES values.
• LIME and ES values are connected; ES values justify a specific local weighting kernel used by LIME.
• DeepLIFT and Layer-wise Relevance Propagation are ES value approximations under linearized or reference-based interpretations; this links their computations to a principled foundation.
• A visualization of ES values as force-like bars provides intuitive understanding of feature contributions to individual predictions and dataset-wide patterns.