[Paper Review] Reproducibility as a Technical Specification

Tom Crick, Benjamin A. Hall+1 more|arXiv (Cornell University)|Apr 6, 2015

Scientific Computing and Data Management53 references3 citations

TL;DR

This paper proposes a formal technical specification for a reproducibility service that enables automated verification and validation of computational research artifacts through standardized cyberinfrastructure and workflows. By integrating artifact evaluation into the publication process—using a tiered review system and community-curated repositories—it establishes a measurable, enforceable standard for reproducibility in computational science, with a case study using the BioModelAnalyzer tool.

ABSTRACT

Reproducibility of computationally-derived scientific discoveries should be a certainty. As the product of several person-years' worth of effort, results -- whether disseminated through academic journals, conferences or exploited through commercial ventures -- should at some level be expected to be repeatable by other researchers. While this stance may appear to be obvious and trivial, a variety of factors often stand in the way of making it commonplace. Whilst there has been detailed cross-disciplinary discussions of the various social, cultural and ideological drivers and (potential) solutions, one factor which has had less focus is the concept of reproducibility as a technical challenge. Specifically, that the definition of an unambiguous and measurable standard of reproducibility would offer a significant benefit to the wider computational science community. In this paper, we propose a high-level technical specification for a service for reproducibility, presenting cyberinfrastructure and associated workflow for a service which would enable such a specification to be verified and validated. In addition to addressing a pressing need for the scientific community, we further speculate on the potential contribution to the wider software development community of services which automate de novo compilation and testing of code from source. We illustrate our proposed specification and workflow by using the BioModelAnalyzer tool as a running example.

Motivation & Objective

• To address the persistent reproducibility crisis in computational science by framing reproducibility as a technical specification rather than a cultural or social issue.
• To design a standardized, measurable, and verifiable service for evaluating research software artifacts, enabling automated compilation and testing from source.
• To integrate artifact evaluation into the academic publishing workflow, starting as optional and evolving into a mandatory requirement over time.
• To reduce barriers for early-career researchers and under-resourced groups by minimizing infrastructure and licensing overheads.
• To establish a community-curated repository of evaluated artifacts to promote best practices and exemplars in reproducible research.

Proposed method

• Design a high-level technical specification for a reproducibility service that supports automated de novo compilation and testing of source code.
• Define a cyberinfrastructure stack with standardized toolchains and workflows to validate artifacts across diverse computational environments.
• Implement a tiered artifact evaluation process using a traffic-light system (e.g., pass, warn, fail) to rate reproducibility levels.
• Introduce a staged adoption model: optional in year t, mandatory without impact on review in year t+1, and mandatory with impact on review in year t+2.
• Use the BioModelAnalyzer tool as a running example to illustrate the specification and workflow in practice.
• Establish a community-driven curation system to maintain a growing, searchable database of evaluated artifacts for benchmarking and comparison.

Experimental results

Research questions

• RQ1 How can reproducibility in computational science be formalized as a measurable, technical specification rather than a cultural ideal?
• RQ2 What cyberinfrastructure and workflow components are required to enable automated, repeatable verification of research software artifacts?
• RQ3 How can artifact evaluation be integrated into the academic publishing process without imposing undue burden on researchers?
• RQ4 What phased adoption strategy would enable widespread community uptake while minimizing disruption?
• RQ5 How can a standardized, community-curated repository of evaluated artifacts promote best practices and improve research quality?

Key findings

• The proposed technical specification enables automated, repeatable verification of computational research artifacts through standardized compilation and testing workflows.
• A staged adoption model—starting optional, then mandatory without review impact, then mandatory with review impact—can facilitate cultural change with minimal disruption.
• The traffic-light system for artifact evaluation provides a clear, visual, and scalable method to communicate reproducibility levels to reviewers and readers.
• Community curation of evaluated artifacts creates a growing, reusable knowledge base that supports benchmarking, comparison, and best practice sharing.
• The framework reduces the risk of 'weaponizing' reproducibility by ensuring consistent, transparent, and uniform evaluation criteria across all submissions.
• The case study with BioModelAnalyzer demonstrates the feasibility and practicality of applying the specification to real-world scientific software.