[Paper Review] Feature Attributions and Counterfactual Explanations Can Be Manipulated

Dylan Slack, Sophie Hilgard+2 more|arXiv (Cornell University)|Jun 23, 2021

Adversarial Robustness in Machine Learning16 references4 citations

TL;DR

This paper demonstrates that feature attribution methods (e.g., LIME, SHAP) and counterfactual explanations (e.g., Wachter’s algorithm, DiCE) can be manipulated by adversaries to hide model biases. By training biased models with a specially designed objective, the authors show that explanations appear fair and unbiased, even when the underlying model systematically disadvantages protected groups.

ABSTRACT

As machine learning models are increasingly used in critical decision-making settings (e.g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions. Such extit{explanations} are used to understand and establish trust in models and are vital components in machine learning pipelines. Though explanations are a critical piece in these systems, there is little understanding about how they are vulnerable to manipulation by adversaries. In this paper, we discuss how two broad classes of explanations are vulnerable to manipulation. We demonstrate how adversaries can design biased models that manipulate model agnostic feature attribution methods (e.g., LIME \& SHAP) and counterfactual explanations that hill-climb during the counterfactual search (e.g., Wachter's Algorithm \& DiCE) into extit{concealing} the model's biases. These vulnerabilities allow an adversary to deploy a biased model, yet explanations will not reveal this bias, thereby deceiving stakeholders into trusting the model. We evaluate the manipulations on real world data sets, including COMPAS and Communities \& Crime, and find explanations can be manipulated in practice.

Motivation & Objective

• To investigate whether post-hoc explanation methods—specifically feature attributions and counterfactual explanations—can be manipulated to conceal model biases.
• To demonstrate that adversaries can train models that appear fair in explanations while being inherently unfair in predictions.
• To evaluate the vulnerability of real-world datasets (COMPAS, Communities & Crime) to such manipulations using state-of-the-art explanation methods.
• To show that counterfactual explanations can be manipulated to produce low-cost recourse for non-protected groups while appearing fair on unperturbed data.

Proposed method

• Design a bi-level optimization objective that jointly minimizes model classification loss, ensures fairness in counterfactual explanations, and enforces small perturbations to the input space.
• Use implicit differentiation to backpropagate through black-box counterfactual algorithms (e.g., Wachter’s, DiCE) during training.
• Train a neural network model with a loss function that includes fairness in counterfactual recourse, unfairness via perturbation-induced cost reduction, and small perturbation constraints.
• Apply a learned perturbation vector δ to non-protected instances to reduce the cost of counterfactuals, while maintaining low disparity in recourse cost on unperturbed data.
• Evaluate the model using both fairness disparity (difference in average recourse cost between protected and non-protected groups) and cost reduction ratio (post-perturbation vs. pre-perturbation cost).
• Use real-world datasets (Communities & Crime, COMPAS) and evaluate with LIME, SHAP, Wachter’s algorithm, Sparse Wachter, DiCE, and prototype-guided counterfactuals.

Experimental results

Research questions

• RQ1 Can adversaries design machine learning models that appear fair in feature attributions (e.g., LIME, SHAP) while being biased in predictions?
• RQ2 Can counterfactual explanation methods be manipulated to produce low-cost recourse for non-protected groups while maintaining apparent fairness on unperturbed data?
• RQ3 To what extent do gradient-hill-climbing counterfactual algorithms (e.g., Wachter’s, DiCE) produce different results under small input perturbations, enabling manipulation?
• RQ4 How effective is the proposed manipulation strategy in practice on real-world datasets with respect to both explanation faithfulness and recourse cost disparity?

Key findings

• The manipulated models achieved within 1% accuracy of baseline models on Communities & Crime, indicating minimal performance degradation.
• On unperturbed data, the disparity in average recourse cost between protected and non-protected groups was minimal (e.g., 0.37 for Wachter’s, 6.99 for DiCE), suggesting apparent fairness.
• After applying the perturbation δ, the cost of recourse for non-protected groups dropped significantly, with cost reduction ratios of up to 20.1× for Wachter’s algorithm.
• For DiCE, the cost reduction was 4.5×, showing that low-cost recourse was easily obtainable for non-protected individuals after manipulation.
• The results demonstrate that counterfactual explanations can be manipulated to hide hidden biases, making models appear fair in explanations while being unfair in practice.
• The manipulation is effective across multiple counterfactual algorithms, including Wachter’s, DiCE, and prototype-guided methods, indicating broad vulnerability.