[Paper Review] Improving Adversarial Robustness via Channel-wise Activation Suppressing

Yang Bai, Yuyuan Zeng+4 more|arXiv (Cornell University)|Mar 11, 2021

Adversarial Robustness in Machine Learning39 references59 citations

TL;DR

The paper reveals two channel-wise activation properties of adversarial examples (higher magnitudes and more uniform channel activation) and introduces Channel-wise Activation Suppressing (CAS) to dynamically suppress redundant channels, enhancing robustness when combined with adversarial training.

ABSTRACT

The study of adversarial examples and their activation has attracted significant attention for secure and robust learning with deep neural networks (DNNs). Different from existing works, in this paper, we highlight two new characteristics of adversarial examples from the channel-wise activation perspective: 1) the activation magnitudes of adversarial examples are higher than that of natural examples; and 2) the channels are activated more uniformly by adversarial examples than natural examples. We find that the state-of-the-art defense adversarial training has addressed the first issue of high activation magnitudes via training on adversarial examples, while the second issue of uniform activation remains. This motivates us to suppress redundant activation from being activated by adversarial perturbations via a Channel-wise Activation Suppressing (CAS) strategy. We show that CAS can train a model that inherently suppresses adversarial activation, and can be easily applied to existing defense methods to further improve their robustness. Our work provides a simple but generic training strategy for robustifying the intermediate layer activation of DNNs.

Motivation & Objective

• Understand how intermediate layer activations relate to adversarial robustness from a channel-wise perspective.
• Identify two activation characteristics of adversarial examples at the channel level (magnitude and activation frequency).
• Propose CAS to suppress redundant channel activations and improve robustness when integrated with existing defenses.

Proposed method

• Analyze channel-wise activations by applying global average pooling to penultimate-layer features.
• Show that adversarial examples have higher channel magnitudes and more uniform activation frequencies than natural examples.
• Introduce CAS: an auxiliary classifier learns channel importance and reweights channel activations during training.
• Integrate CAS with standard adversarial training (and variants like TRADES, MART) with a joint loss.
• Provide an insertion strategy of CAS Modules into intermediate layers and analyze robustness.
• Evaluate robustness against white-box and black-box attacks on CIFAR-10 and SVHN.

Experimental results

Research questions

• RQ1 Do adversarial perturbations cause higher channel-wise activation magnitudes and more uniform channel activation compared to natural inputs?
• RQ2 Can a channel-wise suppression mechanism improve robustness beyond standard adversarial training?
• RQ3 Is CAS effective when applied to different intermediate layers and defense frameworks?
• RQ4 How robust is the CAS module itself under various attacks?
• RQ5 Does CAS maintain or improve natural accuracy while boosting adversarial robustness?

Key findings

• Adversarial examples generally yield higher channel-wise activation magnitudes than natural examples; adversarial training narrows this gap by reducing adversarial magnitudes.
• Adversarial examples activate channels more uniformly, activating many low-frequency channels that natural inputs rarely activate.
• CAS can further improve robustness when added to adversarial training, TRADES, or MART, by suppressing redundant channel activations.
• CAS achieves channel-wise suppression particularly when inserted at deeper blocks (e.g., after Block4 in ResNet-18).
• The CAS-enhanced defenses show improved white-box and black-box robustness on CIFAR-10 and SVHN across FGSM, PGD-20, CW∞, and adaptive attacks.