[Paper Review] Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples

Yinpeng Dong, Fan Bao+2 more|arXiv (Cornell University)|Jan 25, 2019

Adversarial Robustness in Machine LearningComputer Science33 references88 citations

TL;DR

The paper defines neuron consistency metrics, reveals ambiguity in learned features via adversarial examples, and shows an adversarial training approach with a consistent loss improves neuron interpretability while sacrificing some accuracy.

ABSTRACT

Sometimes it is not enough for a DNN to produce an outcome. For example, in applications such as healthcare, users need to understand the rationale of the decisions. Therefore, it is imperative to develop algorithms to learn models with good interpretability (Doshi-Velez 2017). An important factor that leads to the lack of interpretability of DNNs is the ambiguity of neurons, where a neuron may fire for various unrelated concepts. This work aims to increase the interpretability of DNNs on the whole image space by reducing the ambiguity of neurons. In this paper, we make the following contributions: 1) We propose a metric to evaluate the consistency level of neurons in a network quantitatively. 2) We find that the learned features of neurons are ambiguous by leveraging adversarial examples. 3) We propose to improve the consistency of neurons on adversarial example subset by an adversarial training algorithm with a consistent loss.

Motivation & Objective

• Motivate the need for interpretable DNNs in high-stakes applications because individual neurons can be ambiguous in their concept alignment.
• Introduce a quantitative consistency metric for neurons across concepts and assess ambiguity using adversarial samples.
• Propose an adversarial training framework with a consistent (feature-matching) loss to improve neuron consistency on adversarial subsets.

Proposed method

• Define a neuron-concept consistency metric consis(n,c) based on Pr(x contains c | x activates n).
• Compute neuron consistency using a WordNet-based correlation matrix A to produce consis(n)=p^T A p.
• Show that neurons exhibit ambiguity when evaluated on adversarial samples versus real images.
• Propose a training objective with a consistent loss that minimizes classification loss on real and adversarial examples plus a feature-distance term between real and adversarial representations.
• Use FGSM to generate adversarial examples and train with a combined loss L(θ) = αℓ(θ,x,y) + (1−α)ℓ(θ,x*,y) + β d(φθ(x), φθ(x*)) where x* is adversarial and φθ(·) is a feature representation.
• Evaluate on ImageNet and Broden datasets to measure consistency and concept alignment.

Experimental results

Research questions

• RQ1 Do neurons exhibit concept ambiguity when evaluated with adversarial inputs?
• RQ2 Can a consistent adversarial training objective reduce neuron ambiguity on adversarial subsets without severely harming accuracy?
• RQ3 How does adversarial training affect the alignment of neurons with semantic concepts across different architectures?
• RQ4 What is the impact of neuron consistency on interpretability metrics derived from Broden-style concept alignment?
• RQ5 Does enforcing consistency improve interpretability while maintaining competitive robustness?

Key findings

• Adversarial training with a consistent loss increases neuron consistency on adversarial image subsets across architectures.
• Neurons in adversarially trained models maintain more interpretable activations under adversarial inputs than normally trained models.
• On Broden, adversarially trained models show higher alignment of neurons with semantic concepts for both real and adversarial images.
• There is a trade-off: adversarial training reduces standard accuracy by about 10 percentage points but improves robustness to adversarial attacks.
• Models trained with the consistent loss exhibit reduced ambiguity in highest-activation visualizations for adversarial samples (e.g., VGG-16-Adv shows more similar concepts for real and adversarial inputs).
• The proposed metrics quantify neuron-consistency changes and support the claim that interpretability can be enhanced via adversarially guided training.