[Paper Review] Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers

Kaidi Xu, Huan Zhang+5 more|arXiv (Cornell University)|Nov 27, 2020

Adversarial Robustness in Machine Learning36 references59 citations

TL;DR

The paper introduces an optimized LiRPA-based bounding method to enable fast, complete neural network verification within a BaB framework, achieving large speedups over LP-based approaches by leveraging GPUs and batch processing while using limited LP checks to guarantee completeness.

ABSTRACT

Formal verification of neural networks (NNs) is a challenging and important problem. Existing efficient complete solvers typically require the branch-and-bound (BaB) process, which splits the problem domain into sub-domains and solves each sub-domain using faster but weaker incomplete verifiers, such as Linear Programming (LP) on linearly relaxed sub-domains. In this paper, we propose to use the backward mode linear relaxation based perturbation analysis (LiRPA) to replace LP during the BaB process, which can be efficiently implemented on the typical machine learning accelerators such as GPUs and TPUs. However, unlike LP, LiRPA when applied naively can produce much weaker bounds and even cannot check certain conflicts of sub-domains during splitting, making the entire procedure incomplete after BaB. To address these challenges, we apply a fast gradient based bound tightening procedure combined with batch splits and the design of minimal usage of LP bound procedure, enabling us to effectively use LiRPA on the accelerator hardware for the challenging complete NN verification problem and significantly outperform LP-based approaches. On a single GPU, we demonstrate an order of magnitude speedup compared to existing LP-based approaches.

Motivation & Objective

• Motivate the need for complete NN verification in safety-critical settings and analyze limitations of existing BaB with LP solvers.
• Propose using backward LiRPA bounds with gradient-based optimization to replace LP in BaB for speed and parallelism.
• Ensure completeness by integrating minimal LP-based feasibility checks to detect infeasible splits.
• Leverage GPU/TPU acceleration via batch splitting to exploit massive parallelism in LiRPA bounds.
• Demonstrate empirical speedups over LP-based verifiers on standard CIFAR-10 benchmarks.

Proposed method

• Use backward LiRPA bound propagation to obtain linear bounds on NN outputs with respect to inputs.
• Introduce optimized LiRPA with trainable alpha parameters to tighten intermediate and output bounds through gradient descent.
• Employ a batch BaB workflow that splits multiple ReLU nodes in parallel and computes bounds for many sub-domains on GPUs.
• Invoke LP only when necessary to ensure completeness by checking sub-domain feasibility, especially after many splits.
• Adopt batch splitting, parallel LiRPA computations, and a threshold-based fallback to LP to balance speed and completeness.
• Provide a complete verification framework that is sound and complete under the minimal LP feasibility-check regime (Theorem 3.2).

Experimental results

Research questions

• RQ1 Can optimized LiRPA bounds match or surpass LP-based bounds in BaB for complete NN verification?
• RQ2 Does batch LiRPA with minimal LP feasibility checks provide a complete and scalable verification framework on high-dimensional inputs?
• RQ3 What are the trade-offs between speed and completeness when integrating LiRPA with selective LP checks in BaB?
• RQ4 How does batch parallelism on GPUs impact verification performance on CIFAR-10 style benchmarks?
• RQ5 What are the empirical speedups relative to state-of-the-art LP-based and non-LP verifiers?

Key findings

• Optimized LiRPA bounds, enhanced by gradient-based alpha optimization, can outperform LP bounds for complete verification inBaB.
• Batch splitting and GPU-accelerated LiRPA enable significant speedups over LP-based baselines (≈30X on certain benchmarks).
• Completeness is achieved by integrating minimal LP feasibility checks when LiRPA alone cannot resolve splits (Theorem 3.2).
• The framework attains up to 3X speedup over recent state-of-the-art complete verifiers on CIFAR-10 models Base, Wide, and Deep.
• A batch BaB workflow with parallel LiRPA can process multiple sub-domains concurrently, enhancing hardware efficiency.