[Paper Review] Explaining the Explainer: A First Theoretical Analysis of LIME

Damien Garreau, Ulrike von Luxburg|arXiv (Cornell University)|Jan 10, 2020

Explainable Artificial Intelligence (XAI)14 references51 citations

TL;DR

This paper provides the first theoretical analysis of LIME, deriving closed-form surrogate-coefficient expressions for linear targets and revealing conditions under which LIME highlights true features or can miss important ones.

ABSTRACT

Machine learning is used more and more often for sensitive applications, sometimes replacing humans in critical decision-making processes. As such, interpretability of these algorithms is a pressing need. One popular algorithm to provide interpretability is LIME (Local Interpretable Model-Agnostic Explanation). In this paper, we provide the first theoretical analysis of LIME. We derive closed-form expressions for the coefficients of the interpretable model when the function to explain is linear. The good news is that these coefficients are proportional to the gradient of the function to explain: LIME indeed discovers meaningful features. However, our analysis also reveals that poor choices of parameters can lead LIME to miss important features.

Motivation & Objective

• Motivate interpretability of black-box models in sensitive applications.
• Formalize TabularLIME for linear explainers in a tabular data setting.
• Derive closed-form expressions for surrogate coefficients and relate them to the gradient of the explained function.
• Investigate how parameter choices affect feature discovery and surrogate fidelity.

Proposed method

• Model f as a linear function and analyze TabularLIME with Euclidean distance and Gaussian sampling.
• Discretize features using quantile-based bins and encode as binary interpretable features.
• Formulate the surrogate as a weighted least squares problem on the binary features.
• Derive closed-form expressions for surrogate coefficients in terms of the underlying function parameters.
• Provide probabilistic bounds on the estimation error of the surrogate coefficients and the local surrogate error.
• Discuss the impact of bandwidth ν and discretization on coefficient magnitudes and feature selection.

Experimental results

Research questions

• RQ1 Do the LIME surrogate coefficients reflect the gradient of the black-box model at the instance being explained?
• RQ2 Under what conditions can TabularLIME miss important features due to parameter choices?
• RQ3 What is the local approximation error of the LIME surrogate, and how does it behave with increasing sample size?
• RQ4 How do discretization and sampling settings influence the interpretability and robustness of explanations?

Key findings

• The surrogate coefficients are proportional to the partial derivatives of f at the explained instance.
• With enough samples, TabularLIME coefficients concentrate near their theoretical values, reflecting true feature importance.
• Changing a bandwidth parameter can switch off features, causing important features to disappear from the explanation.
• The local surrogate error is typically bounded away from zero unless certain cancellations occur.
• Nonzero coefficients can persist only for coordinates that influence the prediction; irrelevant coordinates tend toward zero.
• The analysis highlights robustness in explanations but also potential instability due to parameter choices.