[Paper Review] Towards Fast Computation of Certified Robustness for ReLU Networks

Tsui-Wei Weng, Huan Zhang+6 more|arXiv (Cornell University)|Apr 25, 2018

Adversarial Robustness in Machine Learning49 references152 citations

TL;DR

The paper introduces two fast algorithms, Fast-Lin and Fast-Lip, to compute certified lower bounds on the minimum adversarial distortion for ReLU networks, offering much faster and tighter guarantees than prior exact methods.

ABSTRACT

Verifying the robustness property of a general Rectified Linear Unit (ReLU) network is an NP-complete problem [Katz, Barrett, Dill, Julian and Kochenderfer CAV17]. Although finding the exact minimum adversarial distortion is hard, giving a certified lower bound of the minimum distortion is possible. Current available methods of computing such a bound are either time-consuming or delivering low quality bounds that are too loose to be useful. In this paper, we exploit the special structure of ReLU networks and provide two computationally efficient algorithms Fast-Lin and Fast-Lip that are able to certify non-trivial lower bounds of minimum distortions, by bounding the ReLU units with appropriate linear functions Fast-Lin, or by bounding the local Lipschitz constant Fast-Lip. Experiments show that (1) our proposed methods deliver bounds close to (the gap is 2-3X) exact minimum distortion found by Reluplex in small MNIST networks while our algorithms are more than 10,000 times faster; (2) our methods deliver similar quality of bounds (the gap is within 35% and usually around 10%; sometimes our bounds are even better) for larger networks compared to the methods based on solving linear programming problems but our algorithms are 33-14,000 times faster; (3) our method is capable of solving large MNIST and CIFAR networks up to 7 layers with more than 10,000 neurons within tens of seconds on a single CPU core. In addition, we show that, in fact, there is no polynomial time algorithm that can approximately find the minimum $\ell_1$ adversarial distortion of a ReLU network with a $0.99\ln n$ approximation ratio unless $\mathsf{NP}$=$\mathsf{P}$, where $n$ is the number of neurons in the network.

Motivation & Objective

• Motivate certified robustness verification for ReLU networks and address NP-hardness of finding exact minimum distortion.
• Provide fast, certified lower bounds on minimum adversarial distortion for practical network sizes.
• Leverage ReLU structure to derive scalable bounding techniques with guarantees.

Proposed method

• Develop Fast-Lin using explicit linear upper and lower bounds for ReLU activations to obtain certified output bounds under bounded perturbations.
• Develop Fast-Lip by upper bounding the local Lipschitz constant to certify a lower bound on the distortion.
• Define activation pattern partitions and construct matrices to express exact analytic bounds across layers.
• Provide closed-form, layer-wise bounds enabling polynomial-time computation, avoiding SMT/MILP solvers.
• Use a binary search on perturbation radius to obtain maximum certified lower bounds from the output bounds.

Experimental results

Research questions

• RQ1 Can we compute non-trivial, certified lower bounds on minimum adversarial distortion for ReLU networks efficiently at scale?
• RQ2 How close can fast bounding methods get to exact minimum distortion compared to Reluplex and LP-based approaches?
• RQ3 What is the theoretical hardness of obtaining tight approximation guarantees for L1 robustness in ReLU networks?
• RQ4 How can we exploit ReLU activation structure to derive scalable, provable robustness certificates?

Key findings

• Fast-Lin yields bounds whose gap to the exact minimum distortion found by Reluplex is 2-3X in small networks, while being over 10,000× faster.
• Fast-Lin provides comparable bound quality to LP-based methods for larger networks, with gaps within 35% and often around 10%, while being 33–14,000× faster.
• Fast-Lip delivers certified lower bounds by bounding the local Lipschitz constant, enabling scalable certification on networks up to 7 layers and over 10,000 neurons in tens of seconds on a CPU.
• The methods enable computing bounds for large MNIST and CIFAR networks, leveraging polynomial-time computation unlike SMT/MILP solvers.