[Paper Review] An analysis of the effects of sharing research data, code, and preprints on citations

Giovanni Colavizza, Lauren Cadwallader+5 more|arXiv (Cornell University)|Apr 24, 2024

Academic Publishing and Open Access5 citations

TL;DR

The study analyzes how Open Science indicators—data sharing, code sharing, and preprint posting—relate to citation counts using a large OA dataset, finding that preprints and online data sharing correlate with higher citations, while code sharing shows no significant effect.

ABSTRACT

Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains. In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122'000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations. We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% on average. However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.

Motivation & Objective

• Assess whether adopting Open Science practices (data sharing, code sharing, preprints) is associated with higher citation counts for publications.
• Quantify the citation impact of each Open Science practice while controlling for publication-, author-, and disciplinary factors.
• Explore how effects vary across disciplines and data-sharing modalities.
• Provide reproducible methods and data to allow replication and extension of findings.

Proposed method

• Use Open Science Indicators (OSI) dataset ( circa 122k publications from PLOS 2018–2023 and a PMC OA Subset comparator).
• Compute publication- and author-level citation metrics using PMC OA Subset as the citation source.
• Model log-transformed publication citations as a function of OSI indicators and a broad set of controls (year, month, n_authors, n_references, h_index_mean, journal, and ANZSRC division dummies).
• Estimate base and full regression models (OLS and robust) with log(n_cit_tot+1) as the dependent variable.
• Include preprint_match, data_shared/location/repositories_data, code_shared/location, and division indicators as key independent variables.
• Report effects in percentage terms via back-transformed coefficients (elasticities).
• Assess robustness across model specifications and time windows (1–3 year citation windows).
• Make data and code openly available for replication.

Experimental results

Research questions

• RQ1 Do Open Science practices (data sharing, code sharing, preprints) associate with higher citation counts for publications, after controlling for confounds?
• RQ2 How do the effects of data sharing, code sharing, and preprints vary across disciplines and data-sharing modalities?
• RQ3 Are there cumulative effects when multiple Open Science practices are adopted?
• RQ4 What are the limitations and generalizability of these findings beyond the PLOS/Open Access context?

Key findings

• Preprints are associated with a significant positive citation advantage of about 20.2% (±0.7).
• Sharing data in an online repository is associated with a positive citation advantage of about 4.3% (±0.8).
• Sharing code does not yield a statistically significant citation advantage in this sample.
• The effects are cumulative: a paper with both a preprint and online data sharing shows about a 24.5% increase in citations.
• Disciplinary variation is evident, with notable differences in the magnitude and presence of effects across divisions.
• The model explains a substantial portion of variance (R2 around 0.426 in the full model).