[Paper Review] Adversarial attacks and defenses in explainable artificial intelligence: A survey

Hubert Baniecki, Przemysław Biecek|arXiv (Cornell University)|Jun 6, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This survey provides a comprehensive analysis of adversarial attacks on explainable AI (XAI) methods and their defenses, introducing a unified taxonomy and notation to systematize research in adversarial XAI (AdvXAI). It identifies critical vulnerabilities in explanation methods, evaluates defense strategies, and outlines future research directions to enhance the robustness and trustworthiness of XAI in high-stakes applications.

ABSTRACT

Explainable artificial intelligence (XAI) methods are portrayed as a remedy for debugging and trusting statistical and deep learning models, as well as interpreting their predictions. However, recent advances in adversarial machine learning (AdvML) highlight the limitations and vulnerabilities of state-of-the-art explanation methods, putting their security and trustworthiness into question. The possibility of manipulating, fooling or fairwashing evidence of the model's reasoning has detrimental consequences when applied in high-stakes decision-making and knowledge discovery. This survey provides a comprehensive overview of research concerning adversarial attacks on explanations of machine learning models, as well as fairness metrics. We introduce a unified notation and taxonomy of methods facilitating a common ground for researchers and practitioners from the intersecting research fields of AdvML and XAI. We discuss how to defend against attacks and design robust interpretation methods. We contribute a list of existing insecurities in XAI and outline the emerging research directions in adversarial XAI (AdvXAI). Future work should address improving explanation methods and evaluation protocols to take into account the reported safety issues.

Motivation & Objective

• To systematize the growing body of research on adversarial attacks targeting explainable AI (XAI) methods and fairness metrics.
• To identify and categorize key failure modes in XAI arising from data and model manipulation, particularly in high-stakes decision-making contexts.
• To propose a unified notation and taxonomy to bridge the gap between adversarial machine learning (AdvML) and XAI research communities.
• To evaluate existing defense mechanisms, including model regularization, focused sampling, and ensemble-based explanation aggregation.
• To highlight emerging challenges in AdvXAI, including XAI-washing, human susceptibility to misleading explanations, and regulatory implications.

Proposed method

• The survey conducts a systematic literature review of over 50 papers from major ML venues (ICML, ICLR, NeurIPS, AAAI, AIj, NMI) and their citation networks to identify core works in adversarial XAI.
• It introduces a unified taxonomy and notation for adversarial attacks on XAI, distinguishing between data-level, model-level, and explanation-level attacks.
• The paper classifies attacks based on their target (e.g., feature importance, counterfactuals, saliency maps) and access level (white-box, black-box, gray-box).
• Defensive strategies are analyzed, including model regularization, data filtering, and ensemble explanation methods that reduce susceptibility to single-method manipulation.
• The study compares the effectiveness of patching explanation methods versus improving model robustness through training-time defenses.
• It evaluates the impact of human factors, such as information overload and susceptibility to deceptive explanations, in real-world deployment contexts.

Figure 1: Explainable AI methods have vurlnerabilities related to safety and security, which we call failure modes . They mainly exist on two levels exploiting manipulation of data or the model. Adversarial manipulation of explanations leads to misinterpretation of the model’s behaviour in audits, d

Experimental results

Research questions

• RQ1 How do adversarial attacks compromise the trustworthiness and security of XAI methods, particularly in high-stakes domains like judicial decisions or medical diagnostics?
• RQ2 What are the primary failure modes of XAI when subjected to adversarial manipulation of data or model parameters?
• RQ3 How effective are current defense mechanisms—such as model regularization, data filtering, or explanation ensembles—in mitigating adversarial threats to explanations?
• RQ4 In what ways do attacks on fairness metrics (e.g., demographic parity, equal opportunity) intersect with attacks on explanation methods, and what are the implications for ethical AI?
• RQ5 What are the key research gaps and future directions in adversarial XAI, particularly regarding certification, real-world incident tracking, and regulatory compliance?

Key findings

• Adversarial attacks can manipulate XAI outputs—such as saliency maps or feature importance—without altering model predictions, undermining trust in model explanations.
• Black-box attacks on explanation methods are feasible even without model access, using optimization techniques that exploit the structure of explanation algorithms.
• Ensemble-based explanation methods show increased robustness to adversarial perturbations, as attackers typically target a single explanation method.
• Model regularization and focused data sampling are effective defense strategies that improve the robustness of both models and their explanations.
• There is a growing risk of 'XAI-washing', where organizations falsely claim to use authentic, trustworthy explanations to meet regulatory or legal requirements.
• Human stakeholders are vulnerable to manipulation through misleading or excessive explanations, highlighting the need for human-centered evaluation of XAI robustness.

Figure 2: Adversarial example is the most common attack on local explanations of the image classifier’s prediction. Left [adapted from 40 ] : An original image is classified as a “dog” and its explanation points out to features influencing this decision. The image can be adversarially changed with p