[Paper Review] "Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations

Yujia Zhang, Kuangyan Song+3 more|arXiv (Cornell University)|Apr 29, 2019

Explainable Artificial Intelligence (XAI)28 references69 citations

TL;DR

The paper identifies three sources of uncertainty in LIME explanations—sampling randomness, proximity-based variation, and cross-point credibility—demonstrating instability across data points and datasets, including synthetic data, 20 Newsgroups, and COMPAS mimic data.

ABSTRACT

Methods for interpreting machine learning black-box models increase the outcomes' transparency and in turn generates insight into the reliability and fairness of the algorithms. However, the interpretations themselves could contain significant uncertainty that undermines the trust in the outcomes and raises concern about the model's reliability. Focusing on the method "Local Interpretable Model-agnostic Explanations" (LIME), we demonstrate the presence of two sources of uncertainty, namely the randomness in its sampling procedure and the variation of interpretation quality across different input data points. Such uncertainty is present even in models with high training and test accuracy. We apply LIME to synthetic data and two public data sets, text classification in 20 Newsgroup and recidivism risk-scoring in COMPAS, to support our argument.

Motivation & Objective

• Identify and categorize sources of uncertainty in LIME explanations.
• Demonstrate how sampling and proximity affect LIME feature selection.
• Show variability of explanations across different data points and datasets.
• Discuss implications for trust and reliability of post-hoc explanations.

Proposed method

• Define three sources of uncertainty in LIME explanations: sampling variance, proximity sensitivity, and cross-point credibility variation.
• Use synthetic tree-generated data to study sampling variance and proximity effects.
• Apply LIME to text classification on 20 Newsgroups to assess local explanation plausibility across documents.
• Analyze COMPAS mimic model to examine consistency of LIME explanations and potential fairness signals.

Experimental results

Research questions

• RQ1 What are the main sources of uncertainty in LIME explanations?
• RQ2 How does sampling variance affect the stability of LIME feature selections for a given input?
• RQ3 How do sampling proximity and parameter choices influence LIME explanations?
• RQ4 Do LIME explanations exhibit variation in credibility across different data points, and what does this imply for trustworthiness?
• RQ5 Are LIME explanations consistent and trustworthy on real-world datasets like COMPAS mimic data?

Key findings

• Sampling variance causes different features to be selected across repeated LIME trials for a single data point.
• Smaller sampling proximity tends to capture locally important features, while larger proximity captures more global features.
• Explanations vary across data points, leading to varying model credibility in local explanations.
• In the 20 Newsgroups text classification, some top features are meaningful in one document but uninformative in another, indicating inconsistent credibility.
• On the COMPAS mimic dataset, explanations were relatively consistent across data points and trials, suggesting potential trustworthiness in that case and highlighting demographic features as influential.