[Paper Review] Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability

Kai Xiao, Vincent Tjeng+2 more|arXiv (Cornell University)|Sep 9, 2018

Adversarial Robustness in Machine Learning33 references64 citations

TL;DR

The paper introduces co-design Training with weight sparsity and ReLU stability (RS Loss) to make adversarial robustness verifiable faster, achieving 4–13x speedups and improved provable accuracy on MNIST and CIFAR-10. RS Loss plus sparsity techniques yield significant verification speedups with minimal accuracy loss.

ABSTRACT

We explore the concept of co-design in the context of neural network verification. Specifically, we aim to train deep neural networks that not only are robust to adversarial perturbations but also whose robustness can be verified more easily. To this end, we identify two properties of network models - weight sparsity and so-called ReLU stability - that turn out to significantly impact the complexity of the corresponding verification task. We demonstrate that improving weight sparsity alone already enables us to turn computationally intractable verification problems into tractable ones. Then, improving ReLU stability leads to an additional 4-13x speedup in verification times. An important feature of our methodology is its "universality," in the sense that it can be used with a broad range of training procedures and verification approaches.

Motivation & Objective

• Motivate co-design of training and verification to enable faster, exact robustness verification.
• Show that weight sparsity and ReLU stability reduce verifier complexity.
• Develop regularization techniques to induce ReLU stability without large accuracy loss.
• Demonstrate speedups and provable robustness improvements on MNIST and CIFAR-10.
• Provide a universal approach compatible with LP/MILP-based verifiers.

Proposed method

• Use L1 regularization and small weight pruning to increase weight sparsity and reduce verification variables.
• Define ReLU stability and introduce RS Loss to encourage stability via a smooth surrogate (F = -tanh(1 + u·l)).
• Estimate ReLU bounds using interval arithmetic (naive or improved) to compute u and l for RS Loss.
• Train with robust adversarial training (Madry et al. 2018) plus RS Loss and sparsity techniques.
• Evaluate with an exact MILP-based verifier (Tjeng et al. 2019) to measure provable robustness and solve times.
• Compare against baselines and certifiable defenses to assess speedups and accuracy.

Experimental results

Research questions

• RQ1 Can co-design training with verification objectives reduce exact verification complexity?
• RQ2 How do weight sparsity and ReLU stability individually and jointly affect verification speed and provable robustness?
• RQ3 What is the practical accuracy/verification trade-off when enforcing ReLU stability via RS Loss?
• RQ4 Are these methods universal across verifiers and scalable to CIFAR-10?
• RQ5 How do RS Loss and sparsity fare against different perturbation strengths (epsilon) in provable robustness?

Key findings

• Weight sparsity from L1 regularization and pruning dramatically improves verification speed and provable robustness on MNIST (e.g., 89.13% provable adversarial accuracy at epsilon=0.1 with pruning).
• RS Loss reduces unstable ReLUs and speeds up verification by 4–13x on MNIST for epsilon in {0.1,0.2,0.3}, with small accuracy loss.
• For CIFAR-10, RS Loss yields smaller speedups (1.6–3.7x) but still improves provable robustness for some settings.
• Combining sparsity and RS Loss maintains state-of-the-art accuracy while enabling significantly faster exact verification.
• +RS networks achieve higher Provable Adversarial Accuracy across tested epsilons (e.g., MNIST epsilon=0.1 from 91.58% to 94.33%).
• The approach demonstrates improved provable robustness on CIFAR-10 and scales to larger networks with maintained efficiency.