[Paper Review] Interpretations Cannot Be Trusted: Stealthy and Effective Adversarial Perturbations against Interpretable Deep Learning

Eldor Abdukhamidov, Mohammed Abuhamad+3 more|arXiv (Cornell University)|Nov 29, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes AdvEdge and AdvEdge+, two stealthy adversarial attacks that simultaneously fool both deep learning classifiers and their coupled interpretation models by exploiting edge information in images. The attacks achieve high success rates with minimal perturbation and significantly outperform existing methods in transferability, demonstrating that interpretability offers limited security in deep learning systems.

ABSTRACT

Deep learning methods have gained increased attention in various applications due to their outstanding performance. For exploring how this high performance relates to the proper use of data artifacts and the accurate problem formulation of a given task, interpretation models have become a crucial component in developing deep learning-based systems. Interpretation models enable the understanding of the inner workings of deep learning models and offer a sense of security in detecting the misuse of artifacts in the input data. Similar to prediction models, interpretation models are also susceptible to adversarial inputs. This work introduces two attacks, AdvEdge and AdvEdge$^{+}$, that deceive both the target deep learning model and the coupled interpretation model. We assess the effectiveness of proposed attacks against two deep learning model architectures coupled with four interpretation models that represent different categories of interpretation models. Our experiments include the attack implementation using various attack frameworks. We also explore the potential countermeasures against such attacks. Our analysis shows the effectiveness of our attacks in terms of deceiving the deep learning models and their interpreters, and highlights insights to improve and circumvent the attacks.

Motivation & Objective

• To investigate whether interpretability in deep learning models can be exploited to undermine their security.
• To develop adversarial attacks that simultaneously deceive both the deep learning classifier and its interpretation model.
• To evaluate the effectiveness and transferability of such attacks across different architectures, interpretation models, and attack frameworks.
• To explore countermeasures against these novel attacks, including detection and robust training techniques.

Proposed method

• The attacks use edge-aware perturbations to embed adversarial noise in regions highlighted by interpretation models, enhancing stealth and effectiveness.
• AdvEdge and AdvEdge+ are optimized variants of the ADV2 attack, leveraging edge information to minimize perceptible changes while maximizing attack success.
• The method integrates with multiple attack frameworks—PGD, C&W, and StAdv—enabling flexible and robust adversarial example generation.
• Interpretation-based detection is implemented by stacking attribution maps from multiple interpreters (e.g., Grad, CAM, RTS, MASK) into multi-channel inputs to improve adversarial sample detection.
• Robust training is applied using a min-max optimization that minimizes the worst-case L1 interpretation difference between benign and adversarial samples, enhancing model resilience.
• Experiments are conducted on ImageNet and CIFAR-10 using ResNet and DenseNet models with four interpretation models: Grad, CAM, RTS, and MASK.

Experimental results

Research questions

• RQ1 Can adversarial examples be crafted to simultaneously deceive both the deep learning classifier and its interpretation model?
• RQ2 How does edge-aware perturbation improve the stealth and effectiveness of adversarial attacks in interpretable deep learning systems?
• RQ3 What is the transferability of adversarial examples generated by AdvEdge and AdvEdge+ across different interpretation models and attack frameworks?
• RQ4 To what extent does interpretability provide meaningful security in deep learning systems?
• RQ5 What countermeasures can effectively defend against these dual-model adversarial attacks?

Key findings

• AdvEdge and AdvEdge+ achieve high attack success rates—over 90% on ImageNet—while maintaining minimal perturbation and high stealth.
• The attacks demonstrate significantly better transferability across different interpretation models compared to existing methods like ADV2.
• Detection based on multi-interpreter attribution stacking improved accuracy to 0.975 when combining Grad, CAM, and MASK interpreters.
• Interpretation-based robust training with Wide-ResNet on CIFAR-10 maintained 0.685 test accuracy under ε=0 and 0.782 Kendall’s Tau correlation for attribution map similarity under adversarial perturbations.
• The normal-trained model showed a drop in interpretation similarity to -0.183 at ε=10/255, while the adversarially trained model maintained high robustness at 0.676.
• The results confirm that interpretability offers limited security, as adversarial examples can evade detection even with human and interpretation-based scrutiny.