[Paper Review] Reproduction Report for SV-COMP 2023

Marcus Gerhold, Arnd Hartmanns|arXiv (Cornell University)|Mar 11, 2023

Software Engineering Research4 citations

TL;DR

This paper reports a reproducibility study of SV-COMP 2023, a large-scale software verification competition. Using the competition's official artifacts and infrastructure, the authors re-ran a representative subset of benchmarks and found minor score variations due to runtime fluctuations, but no changes in tool rankings—confirming the competition's results as reproducible under the same experimental setup.

ABSTRACT

The Competition on Software Verification (SV-COMP) is a large computational experiment benchmarking many different software verification tools on a vast collection of C and Java benchmarks. Such experimental research should be reproducible by researchers independent from the team that performed the original experiments. In this reproduction report, we present our recent attempt at reproducing SV-COMP 2023: We chose a meaningful subset of the competition and re-ran it on the competition organiser's infrastructure, using the scripts and tools provided in the competition's archived artifacts. We see minor differences in tool scores that appear explainable by the interaction of small runtime fluctuations with the competition's scoring rules, and successfully reproduce the overall ranking within our chosen subset. Overall, we consider SV-COMP 2023 to be reproducible.

Motivation & Objective

• To assess the reproducibility of SV-COMP 2023, a large-scale software verification competition, using independent execution.
• To evaluate whether the competition's results, particularly tool rankings, can be reliably reproduced by an external team.
• To identify potential sources of score variation and assess their impact on the fairness and trustworthiness of the competition outcomes.
• To evaluate the practical feasibility of reproducing large-scale tool competitions and to provide feedback for improving future reproducibility.
• To investigate whether a spot-check approach on a subset of benchmarks can serve as a valid proxy for assessing the reproducibility of the entire competition.

Proposed method

• Selected a representative subset of SV-COMP 2023 benchmarks and categories for re-execution, focusing on key categories like ConcurrencySafety, JavaOverall, and SoftwareSystems.
• Re-executed the competition using the original scripts, tools, and benchmark sets provided in the official archived artifacts.
• Ran the subset on the same infrastructure (168-machine cluster) and under the same configuration as the original competition, hosted by LMU Munich.
• Compared re-executed tool scores and rankings against the original SV-COMP 2023 results to assess reproducibility.
• Analyzed raw result data (.csv files) to investigate discrepancies in scores, particularly for tools near the ranking boundary.
• Identified that score differences were primarily due to small runtime fluctuations, such as changes between soft and hard timeouts or out-of-memory vs. timeout outcomes on borderline cases.

Experimental results

Research questions

• RQ1 Can a representative subset of SV-COMP 2023 be successfully reproduced using the official artifacts and infrastructure?
• RQ2 To what extent do minor score variations in re-executed results affect the overall tool ranking in SV-COMP 2023?
• RQ3 Are observed score differences explainable by runtime fluctuations or system-level instabilities, particularly for borderline benchmark instances?
• RQ4 What are the practical challenges in reproducing large-scale tool competitions, and how can they be mitigated for future editions?
• RQ5 Is a spot-check approach on a subset sufficient to establish confidence in the reproducibility of large-scale experimental results like SV-COMP?

Key findings

• The authors successfully reproduced a representative subset of SV-COMP 2023 using the competition’s official scripts, tools, and infrastructure, confirming the feasibility of reproducibility under the same conditions.
• Minor score variations were observed—e.g., UAutomizer increased from 2725 to 2733 in ConcurrencySafety, and UTaipan from 2607 to 2613—due to runtime fluctuations on borderline cases.
• Despite small score changes, the overall tool ranking remained unchanged across all categories tested, including the gold-medal winner Deagle (4754 points).
• In the Termination category, a discrepancy was found: VeriFuzz showed no 'correct false' results in the re-execution, suggesting a potential issue in the scripts or execution pipeline, though the cause is under investigation.
• For Symbiotic and Deagle, scores matched exactly between original and re-executed runs, indicating high reproducibility for these tools.
• Small differences in runtime and resource usage were observed across all tools, but these did not affect the final correctness or ranking outcomes.