[Paper Review] The Reactive Synthesis Competition (SYNTCOMP): 2018-2021

Swen Jacobs, Guillermo A. Pérez+23 more|arXiv (Cornell University)|Jun 1, 2022

Machine Learning in Materials ScienceMaterials Science18 citations

TL;DR

This paper presents a comprehensive analysis of the Reactive Synthesis Competition (SYNTCOMP) from 2018 to 2021, evaluating the performance and evolution of synthesis tools across LTL, safety, and parity game specifications. It introduces new benchmark families, reports on tool rankings by solution quality and scalability, and documents measurable improvements in solver performance, particularly for Strix and ltlsynt, while outlining future rule changes to enhance competition relevance.

ABSTRACT

We report on the last four editions of the reactive synthesis competition (SYNTCOMP 2018-2021). We briefly describe the evaluation scheme and the experimental setup of SYNTCOMP. Then, we introduce new benchmark classes that have been added to the SYNTCOMP library and give an overview of the participants of SYNTCOMP. Finally, we present and analyze the results of our experimental evaluations, including a ranking of tools with respect to quantity and quality - that is, the total size in terms of logic and memory elements - of solutions.

Motivation & Objective

• To evaluate the state of the art in reactive synthesis tools through a standardized, independent competition framework.
• To assess improvements in tool scalability, solution quality, and robustness across multiple specification formats (LTL, safety, parity games).
• To expand and curate a growing, publicly available benchmark library to reflect real-world synthesis challenges.
• To identify trends in tool performance and guide future development through empirical analysis of competition results.
• To update competition rules and infrastructure to sustain long-term community engagement and industrial relevance.

Proposed method

• The competition uses standardized benchmark formats and a centralized, publicly hosted repository (GitHub) with CC-BY licensed entries.
• Tools are evaluated on realizable and unrealizable benchmarks using wall-clock and user-CPU time, with output size measured in AND-gates for solution quality.
• Cactus plots are used to visualize performance across increasing numbers of solved benchmarks, with logarithmic scaling for time and output size.
• The evaluation includes parameterized benchmark families (e.g., mux, shift, arbiter) to assess scalability on structured, complex specifications.
• Results are aggregated per track (LTL, safety, parity) and analyzed for ranking consistency, outlier performance, and tool-specific strengths.
• Future rule changes include merging sequential and parallel tracks, increasing time limits to 10,000s wall-clock and 40,000s user-CPU time, and continued use of StarExec for execution.

Experimental results

Research questions

• RQ1 How have the performance and solution quality of reactive synthesis tools evolved from 2018 to 2021?
• RQ2 Which benchmark families pose the greatest challenges to current synthesis tools, and how do tool rankings vary across them?
• RQ3 To what extent do new specification formats, such as parity games, enhance tool development and competition diversity?
• RQ4 How does the scalability of tools vary across parameterized families like mux, shift, and arbiter specifications?
• RQ5 What impact do rule changes—such as time limits and track consolidation—have on tool performance and competition fairness?

Key findings

• Strix and ltlsynt consistently ranked as top performers in the LTL synthesis track, with Strix showing strong performance across diverse benchmark families.
• For the mux and shift benchmark families, non-Strix tools outperformed Strix in solving time, indicating tool-specific strengths on combinatorial logic synthesis.
• The collector_v2 and detector families showed a more than tenfold improvement in solving time for the best tool compared to Strix, highlighting performance gaps in specific problem classes.
• In arbiter families like round_robin_arbiter and full_arbiter, Strix remained top-ranked overall but showed inconsistent dominance, suggesting variability in performance across similar specifications.
• The introduction of the parity game track in 2021 successfully attracted tool developers, with several teams extending their tools to participate in LTL tracks as well.
• The competition’s benchmark library grew significantly, with 26 parameterized LTL families used in 2021, supporting detailed scalability and performance analysis.