[Paper Review] Interpretable Deep Learning under Fire

Xinyang Zhang, Ningfei Wang+4 more|arXiv (Cornell University)|Dec 3, 2018

Adversarial Robustness in Machine Learning70 references17 citations

TL;DR

This paper introduces Adv2, a novel adversarial attack framework that simultaneously manipulates both deep neural network (DNN) predictions and their associated interpretation models in interpretable deep learning systems (IDLSes). The study demonstrates that existing IDLSes are highly vulnerable to such attacks, as adversaries can arbitrarily control both the model's output and its explanation, undermining the security assurances provided by interpretability. The key contribution is the identification of the 'prediction-interpretation gap' as a root cause of this vulnerability, along with proposed countermeasures like adversarial interpretation distillation (Aid).

ABSTRACT

Providing explanations for deep neural network (DNN) models is crucial for their use in security-sensitive domains. A plethora of interpretation models have been proposed to help users understand the inner workings of DNNs: how does a DNN arrive at a specific decision for a given input? The improved interpretability is believed to offer a sense of security by involving human in the decision-making process. Yet, due to its data-driven nature, the interpretability itself is potentially susceptible to malicious manipulations, about which little is known thus far. Here we bridge this gap by conducting the first systematic study on the security of interpretable deep learning systems (IDLSes). We show that existing \imlses are highly vulnerable to adversarial manipulations. Specifically, we present ADV^2, a new class of attacks that generate adversarial inputs not only misleading target DNNs but also deceiving their coupled interpretation models. Through empirical evaluation against four major types of IDLSes on benchmark datasets and in security-critical applications (e.g., skin cancer diagnosis), we demonstrate that with ADV^2 the adversary is able to arbitrarily designate an input's prediction and interpretation. Further, with both analytical and empirical evidence, we identify the prediction-interpretation gap as one root cause of this vulnerability -- a DNN and its interpretation model are often misaligned, resulting in the possibility of exploiting both models simultaneously. Finally, we explore potential countermeasures against ADV^2, including leveraging its low transferability and incorporating it in an adversarial training framework. Our findings shed light on designing and operating IDLSes in a more secure and informative fashion, leading to several promising research directions.

Motivation & Objective

• To investigate the security vulnerabilities of interpretable deep learning systems (IDLSes), where both the DNN classifier and its interpretation model are susceptible to adversarial manipulation.
• To address the critical gap in understanding whether interpretability, often seen as a security enabler, can be subverted by adversarial attacks.
• To identify the root causes of vulnerability in IDLSes, particularly the misalignment between DNN predictions and interpretation model outputs.
• To evaluate the transferability of adversarial inputs across different interpretation models and explore ensemble-based defenses.
• To propose and validate a novel adversarial training framework, adversarial interpretation distillation (Aid), to improve robustness of interpretation models.

Proposed method

• Propose Adv2, a new class of adversarial attacks that generate inputs which simultaneously mislead the target DNN and deceive its coupled interpretation model.
• Design a joint optimization objective that controls both the DNN’s predicted class and the interpretation model’s attribution map to match the adversary’s desired outcome.
• Empirically evaluate Adv2 on four major types of interpretation models: gradient-based (e.g., Grad-CAM), activation-based (e.g., GradCAM++), perturbation-based (e.g., LIME), and representation-based (e.g., LayerCAM).
• Analyze the prediction-interpretation gap by measuring the statistical and spatial misalignment between DNN predictions and interpretation maps across different models and datasets.
• Investigate transferability of Adv2 across different interpretation models by generating adversarial inputs on one interpreter and testing their effectiveness on others.
• Propose adversarial interpretation distillation (Aid), an adversarial training framework that integrates Adv2-generated examples during interpreter training to improve robustness.

Experimental results

Research questions

• RQ1 Can adversarial inputs be crafted to simultaneously manipulate both the prediction of a DNN and the interpretation generated by its associated interpreter?
• RQ2 What is the role of the prediction-interpretation gap in enabling such dual manipulation, and how does it vary across different interpretation models?
• RQ3 How transferable are Adv2-generated adversarial inputs across different types of interpretation models?
• RQ4 Can adversarial training using Adv2 inputs improve the robustness of interpretation models against such attacks?
• RQ5 To what extent does the current reliance on interpretability in security-critical applications provide a false sense of security?

Key findings

• Adv2 successfully generates adversarial inputs that mislead both the DNN classifier and its interpretation model, allowing the adversary to arbitrarily control both the prediction and the explanation.
• Empirical evaluation on benchmark datasets (e.g., CIFAR-10, ImageNet) and real-world applications (e.g., skin cancer diagnosis) confirms that Adv2 achieves high success rates across diverse DNN and interpretation model combinations.
• The prediction-interpretation gap—where interpretation models do not fully align with DNN decision-making—was identified as a key vulnerability enabling dual manipulation.
• Adv2 exhibits low transferability across different types of interpretation models, suggesting that models from complementary perspectives (e.g., back-propagation vs. input perturbation) are less likely to be fooled by the same adversarial input.
• Adversarial interpretation distillation (Aid) effectively reduces the prediction-interpretation gap and improves the robustness of interpreters against Adv2, as shown through ablation studies.
• The study reveals that interpretability alone cannot be trusted as a security mechanism in adversarial settings, as attackers can exploit the misalignment between prediction and explanation.