[Paper Review] Scalable Verification of Quantized Neural Networks (Technical Report)

Thomas A. Henzinger, Mathias Lechner+1 more|arXiv (Cornell University)|Dec 15, 2020

Adversarial Robustness in Machine Learning27 references4 citations

TL;DR

This paper proposes a scalable SMT-based verification framework for quantized neural networks that accounts for bit-level rounding errors, which are ignored in standard verification methods. By introducing variable pre-computation, minimal bit-width encoding, and redundancy elimination, the approach achieves up to 1,000x speedup over existing tools on adversarial robustness benchmarks, demonstrating that bit-exact verification is feasible despite its PSPACE-hard complexity.

ABSTRACT

Formal verification of neural networks is an active topic of research, and recent advances have significantly increased the size of the networks that verification tools can handle. However, most methods are designed for verification of an idealized model of the actual network which works over real arithmetic and ignores rounding imprecisions. This idealization is in stark contrast to network quantization, which is a technique that trades numerical precision for computational efficiency and is, therefore, often applied in practice. Neglecting rounding errors of such low-bit quantized neural networks has been shown to lead to wrong conclusions about the network's correctness. Thus, the desired approach for verifying quantized neural networks would be one that takes these rounding errors into account. In this paper, we show that verifying the bit-exact implementation of quantized neural networks with bit-vector specifications is PSPACE-hard, even though verifying idealized real-valued networks and satisfiability of bit-vector specifications alone are each in NP. Furthermore, we explore several practical heuristics toward closing the complexity gap between idealized and bit-exact verification. In particular, we propose three techniques for making SMT-based verification of quantized neural networks more scalable. Our experiments demonstrate that our proposed methods allow a speedup of up to three orders of magnitude over existing approaches.

Motivation & Objective

• To address the gap in formal verification of quantized neural networks, which are widely used in practice but often ignored in verification due to rounding errors.
• To show that verifying quantized neural networks with bit-vector specifications is PSPACE-hard, even though satisfiability in the logic is in NP.
• To close the scalability gap between idealized (real-valued) and bit-exact (quantized) neural network verification.
• To develop practical SMT encoding techniques that reduce encoding size and improve solver performance for quantized networks.
• To demonstrate significant performance improvements on standard adversarial robustness benchmarks.

Proposed method

• Introduces a technique to pre-compute values of certain variables and constraints that are independent of inputs, reducing SMT encoding size.
• Proposes minimal bit-width encoding for variables, significantly shrinking the bit-vector representation compared to prior work.
• Develops a redundancy elimination heuristic that exploits bit-level redundancies in network semantics to simplify encodings.
• Applies abstract interpretation to analyze the reachable value range of quantized networks, guiding more efficient encoding.
• Integrates these techniques into an SMT-based verification pipeline using bit-vector logic.
• Employs a portfolio SMT solver (Boolector) and evaluates performance across multiple benchmarks.

Experimental results

Research questions

• RQ1 Is the verification problem for quantized neural networks with bit-vector specifications more complex than for idealized real-valued networks?
• RQ2 Can SMT-based verification of quantized networks be made scalable despite the PSPACE-hard complexity?
• RQ3 What specific encoding optimizations yield the greatest performance gains in SMT-based verification of quantized networks?
• RQ4 How do bit-level rounding errors affect the correctness of verification results when ignored?
• RQ5 To what extent do individual optimization techniques contribute to the overall performance improvement?

Key findings

• The verification problem for quantized neural networks with bit-vector specifications is proven to be PSPACE-hard, even though the underlying logic satisfiability is in NP.
• The proposed method achieves a speedup of over three orders of magnitude (up to 1,000x) compared to existing tools on adversarial robustness verification benchmarks.
• On the MNIST dataset, the method solved 318 out of 391 instances (81.3%) with a cumulative runtime of 7.9 hours, outperforming all ablation variants.
• The ReLU simplification technique contributed most significantly to performance gains, while bit-width minimization and redundancy elimination had limited impact due to solver heuristics.
• Ablation studies show that disabling any of the three main techniques reduces the number of solved instances, confirming their individual contributions.
• The results demonstrate that bit-exact verification is feasible and practical when combined with targeted encoding optimizations.