[Paper Review] Provable Robustness of ReLU networks via Maximization of Linear Regions

Francesco Croce, Maksym Andriushchenko+1 more|arXiv (Cornell University)|Oct 17, 2018

Adversarial Robustness in Machine Learning72 citations

TL;DR

The paper introduces a Maximum Margin Regularizer (MMR) for ReLU networks that enlarges linear regions and distances to decision boundaries, enabling provable robustness guarantees and improved robustness bounds alongside competitive accuracy.

ABSTRACT

It has been shown that neural network classifiers are not robust. This raises concerns about their usage in safety-critical systems. We propose in this paper a regularization scheme for ReLU networks which provably improves the robustness of the classifier by maximizing the linear regions of the classifier as well as the distance to the decision boundary. Our techniques allow even to find the minimal adversarial perturbation for a fraction of test points for large networks. In the experiments we show that our approach improves upon adversarial training both in terms of lower and upper bounds on the robustness and is comparable or better than the state-of-the-art in terms of test error and robustness.

Motivation & Objective

• Motivate robustness guarantees for neural networks in safety-critical settings.
• Develop a regularization scheme that increases the size of linear regions and the distance to decision boundaries in ReLU networks.
• Provide computable lower and upper robustness bounds and integrate with adversarial training.
• Demonstrate improved provable robustness and verifiability across multiple datasets and architectures.

Proposed method

• Represent ReLU networks as continuous piecewise affine functions and describe their linear regions Q(x).
• Define distances to region boundaries d_B(x) and to decision boundaries d_D(x) using region-specific affine mappings V^{(l)} and a^{(l)}.
• Derive robustness guarantees: d_B(x) is a lower bound on minimal perturbation when d_B(x) ≤ d_D(x); d_D(x) equals the minimal perturbation when d_D(x) ≤ d_B(x).
• Introduce the Maximum Margin Regularizer (MMR) combining penalties for closeness to region boundaries and to the decision boundary as in equation (5).
• Provide a practical variant kMMR that averages over the k closest region and decision hyperplanes to accelerate training.
• Train with the standard cross-entropy loss plus λ·MMR(x) to obtain provably robust classifiers.
• Argue that MMR improves verifiability by producing models amenable to mixed-integer programming certification.

Experimental results

Research questions

• RQ1 How can we quantify robustness of ReLU networks in terms of distances to linear region boundaries and decision boundaries?
• RQ2 Can regularizing with these geometric distances yield provable lower/upper bounds on adversarial perturbations?
• RQ3 Does Maximium Margin Regularization improve both empirical robustness and certifiability (verifiability) of networks?
• RQ4 How does MMR interact with adversarial training to enhance robustness guarantees across common norms (l2, l∞)?

Key findings

• MMR substantially increases the size of linear regions in trained networks compared to unregularized baselines.
• The proposed robustness guarantees (Theorem 3.1) provide computable lower and upper bounds on minimal perturbations for many inputs, improving certifiability.
• MMR (and MMR combined with adversarial training) yields tighter provable robustness bounds than several prior methods across MNIST, GTS, Fashion-MNIST, and CIFAR-10.
• MMR models are significantly faster to certify with MIP, yielding near-complete certifiability for many settings, unlike plain or some competing methods.
• Empirically, MMR improves both robustness bounds and test accuracy, and enhances verifiability compared to methods like KW or Xiao et al.