[Paper Review] Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

Dylan Slack, Sophie Hilgard+3 more|arXiv (Cornell University)|Nov 6, 2019

Explainable Artificial Intelligence (XAI)20 references169 citations

TL;DR

The paper shows that perturbation-based post hoc explanations like LIME and SHAP can be fooled by adversarially scaffolded classifiers that behave biased on real data but appear unbiased on perturbed data, effectively hiding discrimination.

ABSTRACT

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Using extensive evaluation with multiple real-world datasets (including COMPAS), we demonstrate how extremely biased (racist) classifiers crafted by our framework can easily fool popular explanation techniques such as LIME and SHAP into generating innocuous explanations which do not reflect the underlying biases.

Motivation & Objective

• Motivate and assess the reliability of post hoc explanations (LIME and SHAP) for detecting biases in black-box models used in high-stakes domains.
• Introduce a framework to construct adversarial classifiers (scaffoldings) that preserve real-data bias while appearing unbiased under perturbed inputs.
• Evaluate the framework on real-world datasets (COMPAS, Communities and Crime, German credit) to show explanations can be fooled.
• Analyze robustness of attacks to perturbation schemes and background distributions used in explanations.

Proposed method

• Describe how LIME and SHAP operate as local, perturbation-based explanations.
• Propose a framework that builds an adversarial classifier e that matches f on the real data distribution X_dist but uses an unbiased ψ on perturbed/out-of-distribution points.
• Train an OOD detector is_OOD to distinguish perturbed from in-distribution points using a labeled dataset built from X and its perturbations to guide the scaffold.
• Construct ψ with synthetic uncorrelated features (and optionally existing features) to control explanations while preserving biased behavior on X_dist.
• Demonstrate that perturbations used by LIME/SHAP can be exploited so that race/gender appears unimportant in explanations while predictions remain biased.
• Evaluate the attack across LIME and SHAP using datasets COMPAS, Communities & Crime, and German Credit, reporting shifts in feature importance rankings.

Experimental results

Research questions

• RQ1 Can post hoc explanations (LIME and SHAP) be manipulated to hide underlying discriminatory biases in black-box models?
• RQ2 How can an adversary exploit perturbation-based explanations to produce explanations that appear innocuous on perturbed data but faithful to biased behavior on real data?
• RQ3 What is the efficacy and robustness of such attacks across different datasets and explanation methods?

Key findings

• Adversarial classifiers can cause LIME explanations to shift important features away from sensitive attributes to uncorrelated features with high fidelity across COMPAS and Communities & Crime, and to Loan%Income in German Credit.
• SHAP explanations can also be fooled, with a high proportion of data points losing the sensitive feature from the top attributions, though SHAP’s local accuracy can distribute attributions when multiple features compete.
• LIME explanations are more vulnerable than SHAP; SHAP requires a more accurate OOD detector to completely hide bias, but can be fooled with reasonable OOD accuracy.
• The attack preserves the original classifier’s predictions on in-distribution data while making explanations appear innocuous on perturbed data.
• Fidelity (e) to f is 100% for LIME across datasets; SHAP shows 75-91% fidelity depending on dataset, indicating strong but dataset-dependent success.