[Paper Review] The State of Play of Reproducibility in Statistics: An Empirical Analysis

Xin Xiong, Ivor Cribben|arXiv (Cornell University)|Jan 1, 2022

Scientific Computing and Data Management4 citations

TL;DR

This empirical study evaluates reproducibility in statistics by attempting to reproduce results from 93 fMRI-based statistical papers published between 2010 and 2021. Only 14 (15.1%) papers were fully reproducible, requiring both accessible, executable code and real data that produced matching results, highlighting a critical gap in reproducibility despite statistics' self-proclaimed openness.

ABSTRACT

Reproducibility, the ability to reproduce the results of published papers or studies using their computer code and data, is a cornerstone of reliable scientific methodology. Studies where results cannot be reproduced by the scientific community should be treated with caution. Over the past decade, the importance of reproducible research has been frequently stressed in a wide range of scientific journals such as <i>Nature</i> and <i>Science</i> and international magazines such as <i>The Economist</i>. However, multiple studies have demonstrated that scientific results are often not reproducible across research areas such as psychology and medicine. Statistics, the science concerned with developing and studying methods for collecting, analyzing, interpreting and presenting empirical data, prides itself on its openness when it comes to sharing both computer code and data. In this article, we examine reproducibility in the field of statistics by attempting to reproduce the results in 93 published papers in prominent journals using functional magnetic resonance imaging (fMRI) data during the 2010–2021 period. Overall, from both the computer code and the data perspective, among all the 93 examined papers, we could only reproduce the results in 14 (15.1%) papers, that is, the papers provide both executable computer code (or software) with the real fMRI data, and our results matched the results in the paper. Finally, we conclude with some author-specific and journal-specific recommendations to improve the research reproducibility in statistics.

Motivation & Objective

• To assess the current state of reproducibility in statistical research, particularly in fMRI studies.
• To investigate the availability and usability of computer code and real data in published statistical papers.
• To identify systemic barriers to reproducibility, such as missing files, software dependencies, and undocumented code.
• To evaluate whether the field of statistics lives up to its reputation for openness in sharing code and data.
• To provide actionable recommendations for authors and journals to improve reproducibility standards in statistical research.

Proposed method

• Selected 93 fMRI-based statistical papers from seven prominent journals published between 2010 and 2021.
• Sought to reproduce results using the provided computer code and real fMRI data, treating reproducibility as the ability to re-run analyses and match published findings.
• Evaluated code for completeness, correctness, and dependency issues, including missing files, broken packages, and unmet software requirements.
• Assessed data availability, including whether raw, preprocessed, or simulated data were provided and if they were usable.
• Classified reproducibility outcomes as fully reproducible, partially reproducible, or not reproducible based on code execution and result matching.
• Analyzed journal-level and author-level patterns in reproducibility to inform policy recommendations.

Experimental results

Research questions

• RQ1 What proportion of statistical fMRI studies published between 2010 and 2021 are fully reproducible using provided code and data?
• RQ2 What are the most common technical and procedural barriers preventing reproducibility in statistical research?
• RQ3 How do journal policies and author practices influence the availability and usability of code and data?
• RQ4 To what extent do missing dependencies, undocumented code, or incomplete data sets undermine reproducibility in statistical publications?
• RQ5 What institutional or policy-level changes could significantly improve reproducibility in statistical research?

Key findings

• Only 14 out of 93 (15.1%) fMRI-based statistical papers were fully reproducible, meaning both executable code and real data were available and produced matching results.
• Among papers providing code, 47 (50.5%) had issues such as missing files, unmet software dependencies, or incorrect results upon execution.
• A significant number of papers relied on non-public or inaccessible data, with 28 papers providing only simulated or partially preprocessed data.
• Common technical failures included missing R functions (e.g., 'dlda' not found), missing C++ header files (e.g., 'bayes_gfl.h'), and unmet system requirements (e.g., 'nvcc: Command not found')
• Despite statistics' reputation for openness, 84.9% of papers failed to meet basic reproducibility standards, indicating a systemic gap between policy and practice.
• Journal-level policies were found to be inconsistent, with no standardized requirements for code and data sharing, suggesting a need for stronger institutional enforcement.