[Paper Review] Feature Denoising for Improving Adversarial Robustness

Cihang Xie, Yuxin Wu+3 more|arXiv (Cornell University)|Dec 9, 2018

Adversarial Robustness in Machine Learning26 references86 citations

TL;DR

The paper introduces feature denoising blocks (based on non-local means and other filters) integrated into CNNs and trained with adversarial training to boost robustness against strong white-box and black-box attacks on ImageNet, achieving state-of-the-art results and winning CAAD 2018 defense.

ABSTRACT

Adversarial attacks to image classification systems present challenges to convolutional networks and opportunities for understanding them. This study suggests that adversarial perturbations on images lead to noise in the features constructed by these networks. Motivated by this observation, we develop new network architectures that increase adversarial robustness by performing feature denoising. Specifically, our networks contain blocks that denoise the features using non-local means or other filters; the entire networks are trained end-to-end. When combined with adversarial training, our feature denoising networks substantially improve the state-of-the-art in adversarial robustness in both white-box and black-box attack settings. On ImageNet, under 10-iteration PGD white-box attacks where prior art has 27.9% accuracy, our method achieves 55.7%; even under extreme 2000-iteration PGD white-box attacks, our method secures 42.6% accuracy. Our method was ranked first in Competition on Adversarial Attacks and Defenses (CAAD) 2018 --- it achieved 50.6% classification accuracy on a secret, ImageNet-like test dataset against 48 unknown attackers, surpassing the runner-up approach by ~10%. Code is available at https://github.com/facebookresearch/ImageNet-Adversarial-Training.

Motivation & Objective

• Motivate that adversarial perturbations induce noise in network features and degrade predictions.
• Develop end-to-end trainable denoising blocks that suppress feature noise in intermediate layers.
• Systematically evaluate robustness under strong white-box and black-box attacks on ImageNet.
• Compare multiple denoising operations and architectural choices to identify effective designs.
• Assess the trade-offs between adversarial robustness and clean accuracy when using denoising blocks.

Proposed method

• Insert denoising blocks after selected residual blocks (after res2, res3, res4, res5) in a ResNet backbone.
• Use denoising operations such as non-local means, bilateral filtering, mean filtering, and median filtering within the blocks.
• Wrap the denoising operation with a 1x1 convolution and a residual connection to fuse denoised features with the input.
• Train end-to-end with adversarial training using PGD (epsilon=16, 30 iterations) on large-scale ImageNet, with distributed 128-GPU setup.
• Identify the best-performing denoising variant as non-local means with Gaussian weighting; perform ablations to study the roles of the 1x1 layer and the residual connection.
• Evaluate under white-box PGD attacks (10–2000 iterations) and black-box CAAD 2017/2018-style attacks, reporting top-1 accuracy and all-or-nothing criteria.

Experimental results

Research questions

• RQ1 Can feature-level denoising reduce adversarially induced noise in intermediate CNN representations?
• RQ2 Which denoising operations (non-local means, bilateral, mean, median) best improve robustness under white-box PGD attacks?
• RQ3 Does integrating denoising blocks with adversarial training yield state-of-the-art robustness on ImageNet under white-box and black-box attacks?
• RQ4 What are the trade-offs between adversarial robustness and clean accuracy when using feature denoising blocks?

Key findings

• Under 10-iteration PGD, ResNet-152 with four denoising blocks achieves 55.7% accuracy versus 52.5% for the baseline.
• Under 2000-iteration PGD, the denoising model achieves 42.6% accuracy vs 39.2% for the baseline.
• In black-box CAAD 2018 defense, the method achieved 50.6% accuracy on a secret ImageNet-like test dataset, winning the defense track by about 10 percentage points over the runner-up.
• Ablation shows removing the 1x1 convolution significantly drops performance (e.g., from 45.5% to 36.8% at 100 iterations when removing the 1x1); removing the residual connection destabilizes training, indicating both components are important.
• Non-local denoising blocks provide robustness benefits beyond local denoising, with Gaussian non-local means performing best among variants.
• In non-adversarial (clean) settings, denoising blocks do not significantly improve accuracy, indicating a robustness-specific benefit.