[Paper Review] Interpreting Adversarial Examples by Activation Promotion and Suppression

Kaidi Xu, Sijia Liu+6 more|arXiv (Cornell University)|Apr 3, 2019

Adversarial Robustness in Machine Learning33 references39 citations

TL;DR

The paper analyzes adversarial perturbations through pixel, image, and network perspectives, revealing a promotion-suppression effect (PSE) on activations and linking perturbations to class-discriminative regions and semantic concepts. It also explores interpretability-driven robustness and neuron masking as defenses.

ABSTRACT

It is widely known that convolutional neural networks (CNNs) are vulnerable to adversarial examples: images with imperceptible perturbations crafted to fool classifiers. However, interpretability of these perturbations is less explored in the literature. This work aims to better understand the roles of adversarial perturbations and provide visual explanations from pixel, image and network perspectives. We show that adversaries have a promotion-suppression effect (PSE) on neurons' activations and can be primarily categorized into three types: i) suppression-dominated perturbations that mainly reduce the classification score of the true label, ii) promotion-dominated perturbations that focus on boosting the confidence of the target label, and iii) balanced perturbations that play a dual role in suppression and promotion. We also provide image-level interpretability of adversarial examples. This links PSE of pixel-level perturbations to class-specific discriminative image regions localized by class activation mapping (Zhou et al. 2016). Further, we examine the adversarial effect through network dissection (Bau et al. 2017), which offers concept-level interpretability of hidden units. We show that there exists a tight connection between the units' sensitivity to adversarial attacks and their interpretability on semantic concepts. Lastly, we provide some new insights from our interpretation to improve the adversarial robustness of networks.

Motivation & Objective

• Study how pixel-level perturbations affect CNN predictions from pixel and image viewpoints.
• Explain the effectiveness of different adversarial attacks using attention- and region-based analyses.
• Investigate how adversarial perturbations influence internal CNN representations and concept detectors.
• Link perturbation patterns to class-discriminative image regions and semantic concepts for robustness insights.

Proposed method

• Define a pixel-level sensitivity measure that aggregates changes in true and target class logits for perturbation regions.
• Introduce promotion-suppression ratio (PSR) to categorize perturbations as suppression-dominated, promotion-dominated, or balance-dominated.
• Use class activation maps (CAM) to interpret how perturbations align with discriminative image regions.
• Apply interpretability scores (IS) based on CAM to quantify perturbation alignment with discriminative regions.
• Employ network dissection to assess how perturbations affect concept detectors and their interpretability.
• Propose refinement strategies using CAM to constrain perturbations and study efficacy.

Experimental results

Research questions

• RQ1 How do pixel-level perturbations promote or suppress CNN activations across the true and target labels?
• RQ2 How can CAM and IS quantify the image-level interpretability of adversarial perturbations?
• RQ3 What is the relationship between adversarial perturbations and internal network concepts as revealed by network dissection?
• RQ4 Can interpretation-guided refinements and neuron masking improve robustness without heavily harming clean accuracy?
• RQ5 How do perturbation patterns relate to discriminative image regions across different attacks and targets?

Key findings

• Adversarial perturbations exhibit a promotion-suppression effect on neuron activations, classifiable as suppression-dominated, promotion-dominated, or balanced.
• CAM-based analysis shows perturbations align with discriminative regions for true and target labels, enabling image-level interpretability of attacks.
• There is a measurable link between pixel perturbations and semantic concepts via network dissection, with sensitive units often being interpretable concepts.
• Interpretability-driven refinements can yield more effective attacks by targeting high-sensitivity regions, while certain CAM-constrained refinements may require larger perturbations.
• Sensitive units to adversarial perturbations tend to correspond to high-interpretability concept detectors, especially in deeper network layers.