[Paper Review] MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples

Nicholas Carlini, David Wagner|arXiv (Cornell University)|Nov 22, 2017

Adversarial Robustness in Machine Learning11 references140 citations

TL;DR

The paper shows that MagNet, Efficient Defenses, and APE-GAN are not robust to adversarial examples; transferable attacks can defeat these defenses with slight distortion increases.

ABSTRACT

MagNet and "Efficient Defenses..." were recently proposed as a defense to adversarial examples. We find that we can construct adversarial examples that defeat these defenses with only a slight increase in distortion.

Motivation & Objective

• Assess robustness of MagNet, Efficient Defenses, and APE-GAN against adversarial examples on MNIST and CIFAR-10.

Proposed method

• Use Carlini and Wagner’s L2 attack to craft transferable adversarial examples against defender copies.
• Train multiple local auto-encoders as detectors/reformers to simulate the defense and enable transfer attacks.
• Compute adversarial objective with classifier loss and detector/reformer losses to maximize transferability.
• Evaluate success rates and L2 distortion for targeted attacks on MNIST and CIFAR-10.
• Analyze why APE-GAN fails by comparing distances before and after reconstruction.

Experimental results

Research questions

• RQ1 Do MagNet, Efficient Defenses, and APE-GAN withstand transferable adversarial examples on MNIST and CIFAR-10?
• RQ2 Can attackers exploit transferability by training their own copies of defenses to defeat the protected model?
• RQ3 What is the distortion required for successful targeted adversarial examples against these defenses?
• RQ4 Why does APE-GAN reconstruction fail to remove adversarial perturbations?

Key findings

• MagNet yields about 99%–100% grey-box attack success with L2 distortions around 2.25 (MNIST) and 0.45 (CIFAR).
• Efficient Defenses against Adversarial Attacks achieve 100% attack success with modest distortions (MNIST around 2.04–2.66; CIFAR around 0.56–0.67).
• APE-GAN also fails to provide robust defense with 100% success and distortions around 2.04–0.72 across MNIST and CIFAR.
• Transferring adversarial examples from a copy of the defense to the actual defended model enables high attack success, highlighting weak grey-box robustness.
• All three defenses offer only modest robustness improvements (roughly 30% increase in distortion on average) versus unsecured models.
• The recovered adversarial examples can be more divergent from the original than the attacker’s perturbations, indicating reconstruction does not reliably neutralize adversarial signals.