[Paper Review] Interpreting Adversarially Trained Convolutional Neural Networks

Tianyuan Zhang, Zhanxing Zhu|ePrints Soton (University of Southampton)|May 23, 2019

Adversarial Robustness in Machine LearningComputer Science75 citations

TL;DR

The paper systematically interprets AT-CNNs, showing they develop more shape-biased representations and are less texture-biased than normally trained CNNs, using salience maps and specialized transformed datasets.

ABSTRACT

We attempt to interpret how adversarially trained convolutional neural networks (AT-CNNs) recognize objects. We design systematic approaches to interpret AT-CNNs in both qualitative and quantitative ways and compare them with normally trained models. Surprisingly, we find that adversarial training alleviates the texture bias of standard CNNs when trained on object recognition tasks, and helps CNNs learn a more shape-biased representation. We validate our hypothesis from two aspects. First, we compare the salience maps of AT-CNNs and standard CNNs on clean images and images under different transformations. The comparison could visually show that the prediction of the two types of CNNs is sensitive to dramatically different types of features. Second, to achieve quantitative verification, we construct additional test datasets that destroy either textures or shapes, such as style-transferred version of clean data, saturated images and patch-shuffled ones, and then evaluate the classification accuracy of AT-CNNs and normal CNNs on these datasets. Our findings shed some light on why AT-CNNs are more robust than those normally trained ones and contribute to a better understanding of adversarial training over CNNs from an interpretation perspective.

Motivation & Objective

• Investigate what AT-CNNs learn and why they are more robust than normally trained CNNs.
• Compare AT-CNNs with standard CNNs using qualitative salience maps on clean and transformed images.
• Quantitatively assess sensitivity to texture- versus shape-preserving distortions through designed test datasets.

Proposed method

• Use salience maps (Grad and SmoothGrad) to visualize feature sensitivities of AT-CNNs vs. standard CNNs on clean and transformed images.
• Construct texture- and shape-distorting test sets (stylized images, saturated images, patch-shuffled images) to quantify generalization gaps.
• Evaluate models on three datasets (CIFAR-10, Tiny ImageNet, Caltech-256) under various adversarial trainings (PGD and FGSM) and strengths.
• Compare robustness and accuracy under adversarial perturbations with an underfitting baseline to control for generalization.

Experimental results

Research questions

• RQ1 Do AT-CNNs rely more on shape information than texture information compared to normally trained CNNs?
• RQ2 How do AT-CNNs and standard CNNs perform when textures or shapes are disrupted while other features remain?
• RQ3 Can salience maps reveal qualitatively different feature sensitivities between AT-CNNs and normal CNNs?
• RQ4 Does adversarial training shift the model’s bias toward long-range structure like shapes or edges?

Key findings

• AT-CNNs produce sparser, contour-focused salience maps, emphasizing shapes/edges more than standard CNNs.
• On stylized (texture-altering) images, AT-CNNs retain higher accuracy than standard CNNs, indicating stronger shape bias.
• Under texture/ saturation distortions, AT-CNNs show less performance degradation, suggesting reduced reliance on local texture information.
• Patch-shuffling (disrupting shapes/edges) greatly reduces AT-CNNs’ confidence and accuracy, while standard CNNs remain relatively robust, demonstrating AT-CNNs’ greater reliance on shapes.
• Across datasets, higher adversarial robustness correlates with greater sensitivity to shape/edge features and reduced texture dependence.
• The study suggests robustness from adversarial training is linked to learning long-range, shape-based representations rather than texture cues.