[Paper Review] On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses

Anish Athalye, Nicholas Carlini|arXiv (Cornell University)|Apr 10, 2018

Adversarial Robustness in Machine Learning7 references119 citations

TL;DR

The paper shows two CVPR 2018 defenses (Pixel Deflection and High-Level Representation Guided Denoiser) fail under white-box attacks, reducing ImageNet accuracy to 0% with 4/255 perturbation and achieving 97% targeted success.

ABSTRACT

Neural networks are known to be vulnerable to adversarial examples. In this note, we evaluate the two white-box defenses that appeared at CVPR 2018 and find they are ineffective: when applying existing techniques, we can reduce the accuracy of the defended models to 0%.

Motivation & Objective

• Assess robustness of two CVPR 2018 white-box defenses against strong adversarial attacks.
• Determine whether Pixel Deflection and HGR are effective under white-box threat models.
• Quantify attack success rates and robustness using standard adversarial attack techniques.

Proposed method

• Apply BPDA to handle non-differentiable pixel deflection and evaluate its robustness against white-box attacks.
• Use PGD to craft untargeted adversarial examples bounded by l_infinity = 4/255.
• Evaluate defended models on ImageNet to measure accuracy under attack and targeted success rate.
• Analyze defenses in the white-box threat model where the attacker knows the defense.

Experimental results

Research questions

• RQ1 Are Pixel Deflection and HGR robust under white-box adversarial attacks?
• RQ2 Can standard white-box attacks reduce the accuracy of defended models on ImageNet to near zero?
• RQ3 What are the targeted attack success rates against these defenses under a small perturbation bound?

Key findings

• Pixel Deflection defense accuracy on defended model reduced to 0% under white-box BPDA attack; targeted success ~97%.
• HGR defense under white-box PGD end-to-end achieves 0% accuracy and 100% targeted attack success.
• Both defenses are not robust to adversarial examples in the white-box threat model.