[Paper Review] Looking Deeper into Tabular LIME

Damien Garreau, Ulrike von Luxburg|arXiv (Cornell University)|Aug 25, 2020

Explainable Artificial Intelligence (XAI)28 references10 citations

TL;DR

This paper provides a theoretical analysis of Tabular LIME, proving that in the large sample limit, its interpretable coefficients can be computed explicitly as functions of model parameters and expectations over the black-box model. It shows LIME correctly identifies relevant features for linear and sparse functions but produces misleading artifacts for partition-based models like CART forests.

ABSTRACT

In this paper, we present a thorough theoretical analysis of the default implementation of LIME in the case of tabular data. We prove that in the large sample limit, the interpretable coefficients provided by Tabular LIME can be computed in an explicit way as a function of the algorithm parameters and some expectation computations related to the black-box model. When the function to explain has some nice algebraic structure (linear, multiplicative, or sparsely depending on a subset of the coordinates), our analysis provides interesting insights into the explanations provided by LIME. These can be applied to a range of machine learning models including Gaussian kernels or CART random forests. As an example, for linear functions we show that LIME has the desirable property to provide explanations that are proportional to the coefficients of the function to explain and to ignore coordinates that are not used by the function to explain. For partition-based regressors, on the other side, we show that LIME produces undesired artifacts that may provide misleading explanations.

Motivation & Objective

• To provide a rigorous theoretical foundation for the default implementation of Tabular LIME in the context of tabular data.
• To investigate whether LIME's explanations are mathematically sound for simple, interpretable models such as linear or multiplicative functions.
• To determine whether LIME provably ignores features not used by the black-box model.
• To identify cases where LIME produces misleading or spurious explanations.
• To guide the design of better interpretability methods by exposing theoretical limitations of existing approaches.

Proposed method

• Derives an explicit formula for the interpretable coefficients of Tabular LIME in the large sample limit using expectation computations over the black-box model.
• Uses a Gaussian kernel-based weighting scheme to define local perturbations around the instance to explain.
• Applies statistical learning theory to compute expectations of weighted model outputs, leveraging independence of features.
• Introduces a normalization constant and uses it to simplify coefficient computation in the linear approximation.
• Analyzes the behavior of LIME under structural assumptions on the black-box function (e.g., linearity, sparsity, multiplicative structure).
• Validates theoretical predictions through empirical experiments, showing close alignment with observed behavior.

Experimental results

Research questions

• RQ1 Does Tabular LIME produce explanations that are mathematically consistent with the true underlying function when the black-box model is linear?
• RQ2 Can Tabular LIME provably ignore features that do not influence the black-box model's output?
• RQ3 What happens to LIME's explanations when the black-box model is a partition-based regressor such as a CART forest?
• RQ4 How do the algorithm's parameters (e.g., bandwidth ν) affect the stability and accuracy of the explanations?
• RQ5 Are there structural assumptions on the black-box model under which LIME's behavior can be analytically characterized?

Key findings

• For linear functions, Tabular LIME produces explanations that are proportional to the true coefficients of the function, demonstrating correctness and consistency.
• When the black-box function depends only on a subset of features, Tabular LIME provably assigns zero coefficients to irrelevant features, confirming feature irrelevance detection.
• For multiplicative or sparsely dependent functions, LIME's explanations remain aligned with the true functional structure under the given assumptions.
• In the case of partition-based models like CART forests, LIME generates artifacts in the explanations, such as non-zero coefficients for features not influencing the prediction.
• The theoretical predictions derived from expectation computations closely match empirical results, validating the analytical framework.
• The analysis reveals that the choice of weighting function significantly impacts explanation quality, suggesting that non-default weights could improve performance for certain models.