[Paper Review] Bridging the Reproducibility Divide: Open Source Software's Role in Standardizing Healthcare AI

John Wu, Zhenbang Wu+1 more|arXiv (Cornell University)|Mar 2, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper analyzes reproducibility in AI for healthcare, revealing heavy use of private data and limited code sharing, and argues for open-source practices and benchmarks to improve trust, safety, and impact.

ABSTRACT

Our analysis of recent AI4H publications reveals that, despite a trend toward utilizing open datasets and sharing modeling code, 74% of AI4H papers still rely on private datasets or do not share their code. This is especially concerning in healthcare applications, where trust is essential. Furthermore, inconsistent and poorly documented data preprocessing pipelines result in variable model performance reports, even for identical tasks and datasets, making it challenging to evaluate the true effectiveness of AI models. Despite the challenges posed by the reproducibility crisis, addressing these issues through open practices offers substantial benefits. For instance, while the reproducibility mandate adds extra effort to research and publication, it significantly enhances the impact of the work. Our analysis shows that papers that used both public datasets and shared code received, on average, 110% more citations than those that do neither--more than doubling the citation count. Given the clear benefits of enhancing reproducibility, it is imperative for the AI4H community to take concrete steps to overcome existing barriers. The community should promote open science practices, establish standardized guidelines for data preprocessing, and develop robust benchmarks. Tackling these challenges through open-source development can improve reproducibility, which is essential for ensuring that AI models are safe, effective, and beneficial for patient care. This approach will help build more trustworthy AI systems that can be integrated into healthcare settings, ultimately contributing to better patient outcomes and advancing the field of medicine.

Motivation & Objective

• Assess the current reproducibility landscape in AI for healthcare (AI4H) as of 2024.
• Quantify reliance on private datasets and lack of code sharing in AI4H publications.
• Evaluate the relationship between reproducibility practices and scholarly impact (citations).
• Propose concrete open-source and benchmarking strategies to improve AI4H reproducibility and transparency.

Proposed method

• Compile a large-scale corpus of AI4H papers from CHIL, ML4H, MLHC, and PubMed (2018–2024).
• Develop automated detectors for public dataset usage, code sharing, and topic classification using keywords, PubMed data, and a medically fine-tuned language model.
• Validate automated detections with manual review of a random sample (30 papers) and report accuracy measures.
• Analyze trends by venue, topic, and affiliation; correlate reproducibility signals with citation counts.

Experimental results

Research questions

• RQ1 What is the current state of technical reproducibility in AI4H papers (private data, code sharing, data preprocessing standardization)?
• RQ2 Do reproducibility practices (public data usage and code sharing) correlate with higher citation impact?
• RQ3 What barriers impede reproducibility in AI4H, and what open-source practices could mitigate them?
• RQ4 How do standardization efforts (e.g., OMOP-CDM, MEDS) relate to reproducibility in AI4H?
• RQ5 What concrete open-source tools, benchmarks, and policies could promote reproducibility in AI4H?

Key findings

• 74% of AI4H papers rely on private datasets or do not share code.
• Papers using both public datasets and shared code receive, on average, 110% more citations than those using neither.
• Private datasets dominate roughly 65–75% of dataset usage from 2018–2024; AI4H conferences use public datasets more than PubMed (about 60–70% vs 25%).
• Code sharing is higher in conference venues than in PubMed papers; PubMed articles show less than 20% code sharing.
• Papers that mention public datasets and share code tend to have higher future citation counts; code sharing correlates with higher citations across topics and affiliations.
• Data preprocessing standardization is limited; adoption of OMOP-CDM and MEDS is incomplete.