[Paper Review] Assessing Reproducibility in Evolutionary Computation: A Case Study using Human- and LLM-based Assessment

Francesca Da Ros, Tarik Zaciragic+3 more|arXiv (Cornell University)|Feb 5, 2026

Scientific Computing and Data Management0 citations

TL;DR

The paper analyzes reproducibility practices in GECCO's Evolutionary Combinatorial Optimization and Metaheuristics (ECOM) track over ten years using a manual checklist and an LLM-based RECAP pipeline, reporting an average reproducibility completeness of 0.62, 36.90% artifact provision, and substantial agreement (kappa 0.67) between automated and human assessments.

ABSTRACT

Reproducibility is an important requirement in evolutionary computation, where results largely depend on computational experiments. In practice, reproducibility relies on how algorithms, experimental protocols, and artifacts are documented and shared. Despite growing awareness, there is still limited empirical evidence on the actual reproducibility levels of published work in the field. In this paper, we study the reproducibility practices in papers published in the Evolutionary Combinatorial Optimization and Metaheuristics track of the Genetic and Evolutionary Computation Conference over a ten-year period. We introduce a structured reproducibility checklist and apply it through a systematic manual assessment of the selected corpus. In addition, we propose RECAP (REproducibility Checklist Automation Pipeline), an LLM-based system that automatically evaluates reproducibility signals from paper text and associated code repositories. Our analysis shows that papers achieve an average completeness score of 0.62, and that 36.90% of them provide additional material beyond the manuscript itself. We demonstrate that automated assessment is feasible: RECAP achieves substantial agreement with human evaluators (Cohen's k of 0.67). Together, these results highlight persistent gaps in reproducibility reporting and suggest that automated tools can effectively support large-scale, systematic monitoring of reproducibility practices.

Motivation & Objective

• Assess the completeness of reproducibility reporting in ECOM papers.
• Identify which reproducibility items are consistently reported or omitted.
• Analyze how reproducibility artifacts are provided, maintained, and accessed over time.
• Evaluate the extent to which automated RECAP assessments align with human judgments.

Proposed method

• Develop a structured reproducibility checklist tailored to EC experiments based on ACM standards.
• Apply the checklist to 168 ECOM papers from 2016–2025 via manual assessment; extract artifacts and evaluate feasibility of rerun.
• Design RECAP, an LLM-based pipeline that uses the checklist to automatically evaluate reproducibility signals from paper text and code repositories.
• Process PDF text, extract artifacts, and run code in a sandbox to assess functional exercisability where feasible.
• Compare automated RECAP results with manual assessments to measure agreement and identify divergence.
• Ground evaluations in ACM artifact guidelines and open-science practices while preserving interpretability.

Figure 1 . BPMN model of the manual assessment of a paper.

Experimental results

Research questions

• RQ1 RQ1 How does the overall reproducibility completeness of ECOM papers vary across years?
• RQ2 RQ2 Which reproducibility items are most consistently reported, and which are systematically omitted?
• RQ3 RQ3 How do ECOM papers provide reproducibility artifacts, and how stable are these practices?
• RQ4 RQ4 Do the best paper candidates exhibit different reproducibility reporting patterns compared to other papers?
• RQ5 RQ5 To what extent does RECAP reproduce manual judgments?

Key findings

• Median paper-level reproducibility completeness is 0.61 with a general upward trend after 2020.
• Only 3 of 168 papers (1.79%) provide sufficient material for full reproduction without contacting authors.
• 36.90% of papers provide at least one reproducibility artifact; artifact provision is time-variant and sometimes links become unavailable.
• Only 15% (25/168) use automated parameter tuning; nearly half report statistical tests, but only 13.69% apply multiple-test corrections.
• Nominated best papers show slightly higher completeness but differences are not statistically significant; artifact provision is higher among nominees and winners.
• RECAP achieves a mean per-paper accuracy of 76.8% and a Cohen’s kappa of 0.67 with human assessments, indicating substantial agreement.

Figure 2 . RECAP system overview. The system processes each paper through a field-by-field evaluation loop. Based on field type, it either uses the paper text directly (Std), retrieves cached best paper website data (BP), or processes linked repositories (Art). Each field is evaluated by an LLM with