[Paper Review] How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods.

Dylan Slack, Sophie Hilgard+3 more|arXiv (Cornell University)|Nov 6, 2019

Explainable Artificial Intelligence (XAI)11 references29 citations

TL;DR

This paper demonstrates that post-hoc explanation methods like LIME and SHAP can be systematically deceived through adversarial scaffolding, where a biased classifier is modified to produce misleadingly fair explanations despite retaining harmful biases. The authors show that even extremely racist classifiers can be hidden from explanation tools, rendering them ineffective for detecting bias in high-stakes AI systems.

ABSTRACT

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real-world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.

Motivation & Objective

• To investigate the reliability of post-hoc explanation methods like LIME and SHAP in detecting model biases.
• To expose vulnerabilities in explanation techniques that rely on input perturbations.
• To develop a framework that allows an adversarial entity to hide model biases while preserving predictive performance.
• To demonstrate that even highly biased classifiers can generate innocuous explanations under LIME and SHAP.
• To challenge the assumption that post-hoc explanations can be trusted as unbiased diagnostics of model behavior.

Proposed method

• The authors introduce a scaffolding technique that modifies a biased classifier to produce inputs that appear representative to explanation methods like LIME and SHAP.
• The scaffolded model is trained to maintain high accuracy on the original data distribution while ensuring that perturbations used in explanation methods yield uniformly distributed, non-discriminatory feature importance scores.
• The method uses adversarial training to align the model’s local explanations with a desired, benign explanation pattern, regardless of the model’s actual decision-making process.
• The framework is applied to real-world datasets, including COMPAS, to simulate high-stakes scenarios like criminal justice risk assessment.
• The approach ensures that the model remains biased in its predictions but appears fair in post-hoc explanations.
• The method leverages the inherent sensitivity of LIME and SHAP to input perturbations, exploiting their reliance on local linear approximations to mask underlying bias.

Experimental results

Research questions

• RQ1 Can adversarial scaffolding be used to hide systematic biases in a classifier while preserving its predictive performance?
• RQ2 To what extent can LIME and SHAP be misled into generating innocuous explanations for a biased model?
• RQ3 How effective is the scaffolding technique in making biased models appear fair under standard explanation tools?
• RQ4 Can the framework be applied to real-world datasets with high-stakes implications, such as criminal justice or healthcare?
• RQ5 What are the limitations of relying on post-hoc explanation methods for auditing model fairness?

Key findings

• The proposed scaffolding technique successfully concealed extreme racial bias in a COMPAS-like classifier while maintaining high predictive accuracy on the original data.
• LIME and SHAP produced uniformly distributed, non-discriminatory feature importance scores for the scaffolded model, falsely indicating fairness.
• Even when the original classifier exhibited clear racial disparities, the post-hoc explanations appeared innocuous and unbiased across multiple evaluation metrics.
• The framework demonstrated that explanation tools cannot be trusted as independent validators of model fairness, as they can be manipulated to hide underlying biases.
• The results show that post-hoc explanation methods are vulnerable to adversarial manipulation, undermining their utility in auditing high-stakes AI systems.