[Paper Review] "Why Should I Trust You?": Explaining the Predictions of Any Classifier

Marco Túlio Ribeiro, Sameer Singh+1 more|arXiv (Cornell University)|Feb 16, 2016

Adversarial Robustness in Machine Learning25 references346 citations

TL;DR

The paper introduces LIME, a model-agnostic method for explaining individual predictions with faithful, locally surrogate interpretable models, and SP-LIME for selecting representative explanations to assess a model globally; it demonstrates fidelity and trust-improving benefits across text and image classifiers, with human and simulated experiments.

ABSTRACT

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.

Motivation & Objective

• Motivate the need for explanations to build trust in predictions and in models deployed in real-world settings.
• Propose LIME to explain any classifier by learning a locally faithful, interpretable model around a prediction.
• Introduce SP-LIME to select a diverse, representative set of explanations for global model trust.
• Demonstrate explanations' utility through simulations and human studies in trust-related tasks.

Proposed method

• Define an interpretable representation of inputs (texts as word presence, images as super-pixels).
• Formulate explanations as a model g in a simple, interpretable family G that approximates the black-box f locally around instance x.
• Minimize a locality-weighted loss L(f,g,πx) plus a complexity penalty Ω(g) to obtain the explanation ξ(x).
• Use perturbations around x′ and a proximity kernel πx to fit the local surrogate to f’s outputs.
• Specialize to sparse linear explanations (g(z′)=w·z′) with L2 loss and an L1-based sparsity step (K-LASSO) for text and images.
• Present a practical algorithm (Algorithm 1) and discuss complexity and interpretability trade-offs.

Experimental results

Research questions

• RQ1 Can explanations faithfully reflect a model’s behavior for individual predictions?
• RQ2 Do explanations help users trust predictions and choose between models?
• RQ3 Can a model-wide understanding be built from a small, non-redundant set of explanations?
• RQ4 Is a model-agnostic explainer capable of explaining diverse models (texts, images, neural nets)?

Key findings

• LIME explanations achieve high fidelity to the underlying model in the local neighborhood (e.g., recall >90% for truly important features on two interpretable classifiers).
• Explanations enable better trust in individual predictions and improve decision-making about model use and forbeding untrustworthy models.
• SP-LIME (submodular pick) selects a diverse, representative set of explanations that improves tasks like model comparison and trust-based selections.
• Qualitative examples show intuitive, human-understandable attributions (e.g., words or super-pixels contributing to a class).
• Simulated and human experiments demonstrate that explanations support tasks such as predicting which classifier generalizes better and guiding feature engineering.