[Paper Review] A Unified View of Piecewise Linear Neural Network Verification

Rudy Bunel, Ilker Turkaslan+3 more|arXiv (Cornell University)|Nov 1, 2017

Adversarial Robustness in Machine LearningComputer Science141 citations

TL;DR

The paper unifies existing PL-NN verification methods under a branch-and-bound framework, introduces benchmarks, and shows new algorithmic improvements yielding about two orders of magnitude speedups over the previous state of the art.

ABSTRACT

The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models. Despite the reputation of learned NN models to behave as black boxes and the theoretical hardness of proving their properties, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure and taking insights from formal methods such as Satisifiability Modulo Theory. These methods are however still far from scaling to realistic neural networks. To facilitate progress on this crucial area, we make two key contributions. First, we present a unified framework that encompasses previous methods. This analysis results in the identification of new methods that combine the strengths of multiple existing approaches, accomplishing a speedup of two orders of magnitude compared to the previous state of the art. Second, we propose a new data set of benchmarks which includes a collection of previously released testcases. We use the benchmark to provide the first experimental comparison of existing algorithms and identify the factors impacting the hardness of verification problems.

Motivation & Objective

• Provide a unified framework that encompasses existing PL-NN verification methods.
• Create and extend a benchmark dataset to compare verification algorithms.
• Identify and validate algorithmic improvements in bounds, branching, and search strategies.
• Demonstrate that combining strengths of multiple approaches speeds up verification.
• Offer practical guidance for improving PL-NN verification performance.

Proposed method

• Reframe PL-NN verification as a global optimization problem via branch-and-bound (BaB).
• Show how existing methods (Reluplex, Planet, MIP, etc.) are special cases within the BaB framework.
• Develop and compare several BaB variants with different splitting and bounding strategies.
• Propose improved bound computations, including layer-wise convex relaxations and tighter relaxations via rebuilding bounds.
• Introduce input-domain branching (BaB-input) and smarter branching (BaBSB) informed by fast dual bounds.
• Provide experimental results on benchmark datasets to quantify efficiency gains.

Experimental results

Research questions

• RQ1 Can a unified branch-and-bound framework encompass prior PL-NN verification methods?
• RQ2 What factors (bounds quality, branching strategy, domain partitioning) most influence verification hardness and runtime?
• RQ3 How do different relaxation techniques compare in tightening bounds for PL-NNs?
• RQ4 What benchmarks reveal the strengths and limitations of SMT-based, MIP-based, and BaB-based verification methods?

Key findings

• The unified BaB framework subsumes prior methods and enables substantial speedups over the previous state of the art (nearly two orders of magnitude in some cases).
• Smart branching and rebuilding tighter relaxations at each step (BaBSB with relu-split) dramatically reduce the number of subdomains explored.
• Reluplex, Planet, and MIPplanet perform very fast on shallower networks, but BaBSB achieves higher success on deeper networks (e.g., ACAS) within practical time budgets.
• Input-domain branching (BaB-input) with fast lower bounds and the Kolter & Wong bound approaches can further enhance pruning efficiency.
• Rebuilding the convex relaxation after each split yields significantly tighter lower bounds, especially for deeper networks.
• The introduced PCAMNIST-based benchmark provides controlled variation of architecture to study verification performance.