[Paper Review] A study of data and label shift in the LIME framework

Amir Hossein Akhavan Rahnama, Henrik Boström|arXiv (Cornell University)|Oct 31, 2019

Explainable Artificial Intelligence (XAI)7 references13 citations

TL;DR

This paper investigates data and label shift in LIME's local surrogate explanation framework, demonstrating that random perturbation of features leads to significant distributional shifts in both input instances and black-box predictions. Using maximum mean discrepancy (MMD) tests, the study shows that these shifts negatively correlate with explanation fidelity, especially as the number of nearest neighbors increases, suggesting a need for improved sampling strategies in LIME.

ABSTRACT

LIME is a popular approach for explaining a black-box prediction through an interpretable model that is trained on instances in the vicinity of the predicted instance. To generate these instances, LIME randomly selects a subset of the non-zero features of the predicted instance. After that, the perturbed instances are fed into the black-box model to obtain labels for these, which are then used for training the interpretable model. In this study, we present a systematic evaluation of the interpretable models that are output by LIME on the two use-cases that were considered in the original paper introducing the approach; text classification and object detection. The investigation shows that the perturbation and labeling phases result in both data and label shift. In addition, we study the correlation between the shift and the fidelity of the interpretable model and show that in certain cases the shift negatively correlates with the fidelity. Based on these findings, it is argued that there is a need for a new sampling approach that mitigates the shift in the LIME's framework.

Motivation & Objective

• To investigate whether LIME's perturbation process induces data and label shift in local surrogate models.
• To evaluate the impact of such shifts on the fidelity of LIME's interpretable models.
• To assess whether the current random sampling strategy in LIME undermines the reliability of explanations.
• To propose a need for new sampling approaches that mitigate distributional shifts in LIME's framework.

Proposed method

• The study uses maximum mean discrepancy (MMD) to measure statistical divergence between distributions of original training instances and LIME-perturbed instances (data shift).
• Label shift is assessed by comparing the distribution of black-box model predictions on original local instances versus perturbed ones using MMD.
• Fidelity is quantified using the inverse of the absolute difference between black-box and surrogate model predictions: $ \mathbb{F}(x,y) = \frac{1}{|f_y(x) - g_y(x)| + 1} $.
• Two case studies are analyzed: text classification on Newsgroups using SVM and object detection on ImageNet using Inception V3.
• The experiments systematically vary the number of nearest neighbors ($n$) used in LIME’s local neighborhood selection to assess sensitivity to $n$.
• Two-sample MMD tests at $\alpha = 0.05$ are used to test the null hypotheses $H_0: P_{X_{\text{knn}}} = P_Z$ and $H_0: P_{F(X_{\text{knn}})} = P_{F(Z)}$.

Experimental results

Research questions

• RQ1 Does the perturbed data distribution in LIME differ significantly from the original training distribution (data shift)?
• RQ2 Do the black-box model predictions on perturbed instances differ from those on original local instances (label shift)?
• RQ3 Is there a correlation between the magnitude of data or label shift and the fidelity of the LIME surrogate model?
• RQ4 How does the number of nearest neighbors ($n$) affect the extent of data and label shift in LIME?

Key findings

• For the Newsgroups text classification task, data shift was detected in 100% of test instances when $n \geq 20$, with MMD values rising from 5.56 ($n=20$) to 87.35 ($n=500$).
• In the same task, label shift was detected in 100% of instances when $n \geq 100$, with MMD values increasing from 11.97 ($n=100$) to 63.06 ($n=500$).
• For the ImageNet object detection task, data shift was detected in 100% of instances at $n \geq 5$, with MMD values rising from 0.64 ($n=5$) to 65.32 ($n=500$).
• Label shift was detected in 100% of ImageNet instances at $n \geq 2$, with MMD values increasing from 1.08 ($n=2$) to 346.30 ($n=500$).
• A negative correlation was observed between MMD divergence (data and label shift) and fidelity: as shift increased, fidelity decreased, particularly for larger $n$.
• The results indicate that LIME’s current random sampling strategy induces substantial distributional shifts that compromise explanation reliability, especially as $n$ increases.