[Paper Review] Practice meets Principle: Tracking Software and Data Citations to Zenodo DOIs

Stephanie van de Sandt, Lars Holm Nielsen+7 more|arXiv (Cornell University)|Nov 1, 2019

Scientific Computing and Data Management21 references5 citations

TL;DR

This study analyzes 5,456 citations to Zenodo-hosted software and data from January 2019, using the Asclepias Broker to assess alignment between current citation practices and the FORCE11 Software Citation Principles. It finds widespread mismatch between recommended and actual citation practices, with 82% of citations to software being self-citations, and recommends practical steps for improving citation implementation based on DOI-based versioning and concept DOIs to better track impact.

ABSTRACT

Data and software citations are crucial for the transparency of research results and for the transmission of credit. But they are hard to track, because of the absence of a common citation standard. As a consequence, the FORCE11 recently proposed data and software citation principles as guidance for authors. Zenodo is recognized for the implementation of DOIs for software on a large scale. The minting of complementary DOIs for the version and concept allows measuring the impact of dynamic software. This article investigates characteristics of 5,456 citations to Zenodo data and software that were captured by the Asclepias Broker in January 2019. We analyzed the current state of data and software citation practices and the quality of software citation recommendations with regard to the impact of recent standardization efforts. Our findings prove that current citation practices and recommendations do not match proposed citation standards. We consequently suggest practical first steps towards the implementation of the software citation principles

Motivation & Objective

• To evaluate the alignment between actual software and data citation practices and the FORCE11 Software Citation Principles.
• To assess the impact of recent standardization efforts on citation quality and tracking accuracy.
• To investigate the role of Zenodo’s concept DOI and version DOI system in aggregating citation impact across software versions.
• To identify discrepancies between recommended citation formats and real-world citation behavior in scholarly publications.
• To provide actionable recommendations for improving software citation practices based on empirical citation data.

Proposed method

• Harvested 5,456 citation links to Zenodo objects via the Asclepias Broker, which aggregates data from ADS, Europe PMC, and Crossref Event Data using public APIs.
• Used Spearman’s rank correlation to assess relationships between citation counts and usage metrics (unique views, downloads) from Zenodo and GitHub (stars, forks, watchers).
• Identified self-citations by comparing author lists of citing articles and cited Zenodo objects, using a representative sample of 352 citations with citation speeds of 0 or 1.
• Visualized citation overlap across discovery services (ADS, Europe PMC) using Venn diagrams to assess coverage and redundancy.
• Analyzed citation recommendations on GitHub and documentation for 25 high-citation software concepts, tracking changes in recommendation quality over time.
• Leveraged Zenodo’s concept DOI and version DOI infrastructure to track citation distribution across software versions and estimate impact aggregation.

Experimental results

Research questions

• RQ1 To what extent do actual software and data citations align with the FORCE11 Software Citation Principles?
• RQ2 How effective are current citation recommendations in guiding authors toward proper citation practices?
• RQ3 What is the proportion of self-citations in software and data citations, and how does this affect impact measurement?
• RQ4 How well do citation discovery services (e.g., ADS, Europe PMC) cover citations to Zenodo-hosted software and data?
• RQ5 To what extent can concept DOIs improve the aggregation and tracking of citation impact across software versions?

Key findings

• 82% of citations to software hosted on Zenodo were self-citations, indicating a significant mismatch between recommended and actual citation behavior.
• The citation speed for 4,060 citations was 0 or 1, meaning the cited software was cited in the same year or within a year of publication, suggesting high self-citation rates.
• Spearman’s rank correlation revealed weak to moderate correlations between citation counts and usage metrics (e.g., unique views, downloads), indicating limited alignment between usage and citation impact.
• Only 48% of citing articles were found by both ADS and Europe PMC, highlighting coverage gaps in citation discovery services.
• Among 25 high-citation software concepts, 68% had no stable, versioned citation recommendation, and 40% had recommendations that changed across versions, undermining consistency.
• The use of concept DOIs enabled aggregation of citations across versions, but only 36% of cited software versions were linked to the correct concept DOI, limiting impact tracking.