[Paper Review] Piecewise Linear Neural Networks verification: A comparative study

Rudy Bunel, Ilker Turkaslan+3 more|arXiv (Cornell University)|Nov 1, 2017

Adversarial Robustness in Machine LearningComputer Science16 references49 citations

TL;DR

This paper presents a comparative evaluation of verification methods for piecewise linear neural networks using Mixed Integer Programming, Satisfiability Modulo Theory, and a novel Branch-and-Bound approach. It introduces a new benchmark suite and reveals implementation bugs in prior work, enabling more reliable progress in formal verification of deep learning models.

ABSTRACT

The success of Deep Learning and its potential use in many important safety- critical applications has motivated research on formal verification of Neural Network (NN) models. Despite the reputation of learned NN models to behave as black boxes and the theoretical hardness of proving their properties, researchers have been successful in verifying some classes of models by exploiting their piecewise linear structure. Unfortunately, most of these approaches test their algorithms without comparison with other approaches. As a result, the pros and cons of the different algorithms are not well understood. Motivated by the need to accelerate progress in this very important area, we investigate the trade-offs of a number of different approaches based on Mixed Integer Programming, Satisfiability Modulo Theory, as well as a novel method based on the Branch-and-Bound framework. We also propose a new data set of benchmarks, in addition to a collection of pre- viously released testcases that can be used to compare existing methods. Our analysis not only allows a comparison to be made between different strategies, the comparison of results from different solvers also revealed implementation bugs in published methods. We expect that the availability of our benchmark and the analysis of the different approaches will allow researchers to develop and evaluate promising approaches for making progress on this important topic.

Motivation & Objective

• To address the lack of standardized comparison across neural network verification methods.
• To evaluate the trade-offs between different formal verification techniques for piecewise linear neural networks.
• To identify implementation flaws in existing verification approaches through cross-solver comparison.
• To provide a new, publicly available benchmark suite for fair and reproducible evaluation of verification tools.
• To accelerate progress in formal verification of deep learning models by enabling systematic comparison and validation.

Proposed method

• The study evaluates three main verification strategies: Mixed Integer Programming (MIP), Satisfiability Modulo Theory (SMT), and a novel Branch-and-Bound-based method.
• A new benchmark suite is introduced, combining previously released test cases with new, diverse neural network instances.
• Cross-comparison of results across multiple solvers is used to detect inconsistencies and implementation bugs in published methods.
• The evaluation framework enables systematic analysis of performance, scalability, and correctness across different verification techniques.
• The benchmark is designed to be representative of real-world safety-critical applications and supports reproducible research.

Experimental results

Research questions

• RQ1 How do MIP, SMT, and the proposed Branch-and-Bound method compare in terms of verification performance and scalability?
• RQ2 What are the key strengths and weaknesses of each verification approach in practical settings?
• RQ3 To what extent do discrepancies in results across solvers reveal implementation bugs in existing tools?
• RQ4 How effective is the new benchmark suite in enabling fair and reproducible evaluation of verification methods?
• RQ5 Can cross-solver comparison serve as a reliable method for detecting errors in published verification algorithms?

Key findings

• Cross-solver comparison revealed implementation bugs in previously published verification methods, highlighting the need for rigorous validation.
• The proposed Branch-and-Bound-based method showed competitive performance and scalability, particularly on certain network architectures.
• MIP-based approaches demonstrated strong expressiveness but suffered from scalability limitations on larger networks.
• SMT-based methods exhibited good performance on smaller networks but struggled with complex constraints.
• The new benchmark suite effectively exposed inconsistencies and enabled reliable comparison across tools.
• The study underscores the importance of reproducibility and cross-validation in formal neural network verification.