[Paper Review] Truth, Proof, and Reproducibility: There's no counter-attack for the codeless

Charles T. Gray, Ben Marwick|arXiv (Cornell University)|Jul 11, 2019

Data Visualization and Analytics26 references4 citations

TL;DR

This paper argues that computational scientific research must adopt rigorous, proof-like practices—such as automated testing, version control, and reproducible compendia—to achieve scientific certainty in the absence of formal mathematical proofs. By modeling computational work after mathematical proofs, researchers can ensure reproducibility and trustworthiness, even when direct formal proof is unattainable.

ABSTRACT

Current concerns about reproducibility in many research communities can be traced back to a high value placed on empirical reproducibility of the physical details of scientific experiments and observations. For example, the detailed descriptions by 17th century scientist Robert Boyle of his vacuum pump experiments are often held to be the ideal of reproducibility as a cornerstone of scientific practice. Victoria Stodden has claimed that the computer is an analog for Boyle's pump -- another kind of scientific instrument that needs detailed descriptions of how it generates results. In the place of Boyle's hand-written notes, we now expect code in open source programming languages to be available to enable others to reproduce and extend computational experiments. In this paper we show that there is another genealogy for reproducibility, starting at least from Euclid, in the production of proofs in mathematics. Proofs have a distinctive quality of being necessarily reproducible, and are the cornerstone of mathematical science. However, the task of the modern mathematical scientist has drifted from that of blackboard rhetorician, where the craft of proof reigned, to a scientific workflow that now more closely resembles that of an experimental scientist. So, what is proof in modern mathematics? And, if proof is unattainable in other fields, what is due scientific diligence in a computational experimental environment? How do we measure truth in the context of uncertainty? Adopting a manner of Lakatosian conversant conjecture between two mathematicians, we examine how proof informs our practice of computational statistical inquiry. We propose that a reorientation of mathematical science is necessary so that its reproducibility can be readily assessed.

Motivation & Objective

• To address the growing crisis in reproducibility across scientific disciplines by redefining scientific rigor in computational research.
• To argue that computational science, like mathematics, requires a form of 'proof' to ensure certainty and reproducibility despite the absence of formal logical proofs.
• To propose that good enough practices—such as unit testing, version control, and standardized research compendia—can serve as the computational equivalent of mathematical proof.
• To examine how modern research workflows, especially in computational mathematics, have drifted from the blackboard tradition of proof toward experimental, code-based inquiry.
• To advocate for a cultural and methodological shift in mathematical and computational sciences toward systematic, verifiable, and transparent research practices.

Proposed method

• Adopt a Lakatosian conversational framework between two mathematicians to explore the evolution of proof and reproducibility in computational science.
• Use Catherine and Hal from the play *Proof* as a narrative device to contrast traditional mathematical proof with modern computational research practices.
• Analyze real-world data from CRAN task views to assess test coverage and reproducibility practices in R packages, using test size ratio as a proxy for test coverage.
• Propose that unit testing serves as a computational analog to mathematical proof, providing verifiable, repeatable, and transparent validation of algorithms.
• Advocate for the use of tools like rrtools and DOI-issuing repositories to standardize and preserve research compendia, ensuring long-term reproducibility.
• Integrate principles from software engineering—such as version control, stress testing, and automated validation—into the scientific workflow to enhance scientific diligence.

Experimental results

Research questions

• RQ1 How can computational research achieve a level of certainty comparable to mathematical proof, given the absence of formal logical derivation?
• RQ2 To what extent do current computational research practices in fields like statistics and data science reflect the rigor of mathematical proof?
• RQ3 What role do automated testing, version control, and research compendia play in enhancing reproducibility and scientific trustworthiness?
• RQ4 Why do many R packages, even those in curated CRAN task views, lack unit tests, and what does this imply for scientific reliability?
• RQ5 How can communities of practice in computational science adopt 'good enough' but systematic practices to ensure scientific diligence and reduce questionable research practices?

Key findings

• Among 4,105 R packages listed in CRAN task views, 1,524 (37%) contained no unit tests, indicating widespread gaps in reproducibility practices.
• Despite being considered more stable and trustworthy due to curation, the proportion of task view packages with tests has declined over the past decade.
• Test coverage varies significantly across CRAN task views, with Survival having only 23% of packages with tests, while Web Technologies had 66%, suggesting no consistent pattern in testing adoption.
• No clear correlation was found between test presence and package size, number of authors, or centrality (measured by reverse dependencies and imports), indicating inconsistent adoption of testing practices.
• The use of test size ratio as a proxy for test coverage shows wide variation, with no strong trend over time, highlighting the need for better metrics and tooling.
• The authors conclude that while formal proof is unattainable in computational science, rigorous, automated, and transparent practices—such as unit testing and version control—can serve as a practical, defensible standard of scientific certainty.