[Paper Review] The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results

Stanley Bak, Changliu Liu+1 more|arXiv (Cornell University)|Aug 31, 2021

Adversarial Robustness in Machine LearningComputer Science51 references43 citations

TL;DR

A comprehensive report detailing VNN-COMP 2021, including rules, participating tools, benchmarks, results, and lessons learned from a fair, standardized neural network verification competition.

ABSTRACT

This report summarizes the second International Verification of Neural Networks Competition (VNN-COMP 2021), held as a part of the 4th Workshop on Formal Methods for ML-Enabled Autonomous Systems that was collocated with the 33rd International Conference on Computer-Aided Verification (CAV). Twelve teams participated in this competition. The goal of the competition is to provide an objective comparison of the state-of-the-art methods in neural network verification, in terms of scalability and speed. Along this line, we used standard formats (ONNX for neural networks and VNNLIB for specifications), standard hardware (all tools are run by the organizers on AWS), and tool parameters provided by the tool authors. This report summarizes the rules, benchmarks, participating tools, results, and lessons learned from this competition.

Motivation & Objective

• Establish a fair, standardized platform to compare neural network verification tools.
• Assess scalability and speed of state-of-the-art verification methods using ONNX/VNNLIB formats and AWS hardware.
• Provide benchmarks spanning diverse architectures and applications to drive progress in NN verification.
• Summarize lessons learned to guide future iterations of VNN-COMP.

Proposed method

• Standardized inputs (ONNX networks, VNNLIB specifications) and hardware (AWS CPU/GPU) for all tools.
• Defined per-instance and per-benchmark run-time caps and an overhead-correction procedure to ensure fair timing.
• Implemented a scoring scheme with per-instance correctness, type of correctness, and time-based bonuses.
• Collected and analyzed results across twelve participating teams/tools and a fixed set of benchmarks.
• Provided reproducible pipelines and publicly available benchmarks/scripts via GitHub.

Experimental results

Research questions

• RQ1 How do current NN verification tools compare in terms of scalability and speed under standardized conditions?
• RQ2 What are the strengths and limitations of each tool across a diverse benchmark suite?
• RQ3 What lessons can be learned to improve future VNN-COMP iterations and NN verification research?
• RQ4 How do standard formats and controlled hardware impact fair comparison of verification methods?

Key findings

• Twelve tools participated and were evaluated on a common AWS-based platform.
• Benchmarks covered diverse networks and tasks, including ACASXu, CIFAR-10 variants, MNIST, and others (see benchmarks list).
• The competition established a fair, repeatable pipeline with predefined rules, timeouts, and an overhead correction to account for tool startup time.
• Results and benchmarks were reported publicly, enabling objective comparison and future replication.
• The report documents lessons learned and potential improvements for subsequent VNN-COMP iterations.