[Paper Review] Provably Minimally-Distorted Adversarial Examples

Nicholas Carlini, Guy Katz+2 more|arXiv (Cornell University)|Sep 29, 2017

Adversarial Robustness in Machine Learning20 references100 citations

TL;DR

The paper uses formal verification (Reluplex) to construct adversarial examples with provable minimal distortion and evaluates attacks and defenses, showing CW attacks are near-optimal and adversarial training significantly increases robustness on MNIST in small networks.

ABSTRACT

The ability to deploy neural networks in real-world, safety-critical systems is severely limited by the presence of adversarial examples: slightly perturbed inputs that are misclassified by the network. In recent years, several techniques have been proposed for increasing robustness to adversarial examples --- and yet most of these have been quickly shown to be vulnerable to future attacks. For example, over half of the defenses proposed by papers accepted at ICLR 2018 have already been broken. We propose to address this difficulty through formal verification techniques. We show how to construct provably minimally distorted adversarial examples: given an arbitrary neural network and input sample, we can construct adversarial examples which we prove are of minimal distortion. Using this approach, we demonstrate that one of the recent ICLR defense proposals, adversarial retraining, provably succeeds at increasing the distortion required to construct adversarial examples by a factor of 4.2.

Motivation & Objective

• Motivate robust evaluation of defenses beyond empirical results using formal verification.
• Develop a method to produce adversarial examples that are provably minimally distorted under an chosen metric.
• Assess the effectiveness of a modern attack (Carlini & Wagner) against minimal-distortion benchmarks.
• Assess the robustness impact of adversarial training (Madry et al.) using verifiable guarantees on small networks.

Proposed method

• Use Reluplex, a sound and complete solver for networks with ReLUs, to verify the nonexistence of adversarial inputs within a distance delta.
• Encode L1 distances into Reluplex by representing absolute values with ReLUs.
• Perform a binary search over delta to find provably minimally distorted adversarial examples.
• Compare attacker-generated examples (CW, BIM/FGM) against provably minimal ones to assess near-optimality.
• Evaluate adversarial training by verifying robustness improvements on a small MNIST network.
• Discuss limitations to small networks and potential scalability with improved verification tools.

Experimental results

Research questions

• RQ1 How close are iterative attacks (e.g., CW) to the provably minimally distorted adversarial examples?
• RQ2 Does adversarial training (Madry et al.) provably increase robustness against adversarial examples on small networks?
• RQ3 Can Reluplex-based verification extend to L1 distances and beyond L-infinity for measuring distortion?
• RQ4 What insights do provable guarantees provide about current and future attack/defense effectiveness on compact models?

Key findings

• CW attacks yield adversarial examples within 11.6% of the minimally distorted examples under L-infinity on the small MNIST model.
• CW attacks yield adversarial examples within 6.2% of the minimally distorted examples under L1 on the small MNIST model (limited by termination counts).
• Adversarial training increases the distance to minimally distorted adversarial examples by an average of 4.23x on the tested small network (from 0.039 to 0.165 under L-infinity).
• Provable minimal-distortion adversarial examples can be found for various targets, enabling evaluation of defenses under guaranteed guarantees on small networks.
• Reluplex termination rates are higher on adversarially trained networks for both L-infinity (81/90) and L1 (64/90) in the reported experiments.
• The study shows iterative attacks are near-optimal but often miss the true minimum due to local minima in gradient descent.