[Paper Review] Assigning credit to scientific datasets using article citation networks

Tong Zeng, Longfeng Wu+2 more|arXiv (Cornell University)|Jan 16, 2020

Scientific Computing and Data Management55 references4 citations

TL;DR

This paper proposes DataRank, a network flow algorithm that assigns credit to scientific datasets by modeling citation flows differently between publications and datasets. It outperforms alternatives in predicting real-world usage (e.g., GenBank web visits and Figshare downloads), demonstrating that datasets have distinct citation dynamics and can be ranked meaningfully without requiring explicit dataset citations.

ABSTRACT

A citation is a well-established mechanism for connecting scientific artifacts. Citation networks are used by citation analysis for a variety of reasons, prominently to give credit to scientists' work. However, because of current citation practices, scientists tend to cite only publications, leaving out other types of artifacts such as datasets. Datasets then do not get appropriate credit even though they are increasingly reused and experimented with. We develop a network flow measure, called DataRank, aimed at solving this gap. DataRank assigns a relative value to each node in the network based on how citations flow through the graph, differentiating publication and dataset flow rates. We evaluate the quality of DataRank by estimating its accuracy at predicting the usage of real datasets: web visits to GenBank and downloads of Figshare datasets. We show that DataRank is better at predicting this usage compared to alternatives while offering additional interpretable outcomes. We discuss improvements to citation behavior and algorithms to properly track and assign credit to datasets.

Motivation & Objective

• To address the lack of systematic credit assignment to scientific datasets despite their critical role in reproducibility and reuse.
• To develop a method that infers dataset impact from publication citation networks without requiring changes in citation behavior.
• To model the differing obsolescence rates of publications and datasets in citation networks to improve impact estimation.
• To validate the method using real-world datasets from GenBank and Figshare, measuring prediction accuracy against actual usage metrics.
• To provide a scalable, interpretable framework for ranking datasets that supports fair evaluation and policy-making in science.

Proposed method

• The method extends the NetworkFlow algorithm (Walker et al., 2007) by introducing two distinct node types: publications and datasets.
• It models citation flows with different decay rates for publications and datasets, reflecting their distinct obsolescence patterns.
• A random walker traverses the citation network, assigning higher scores to datasets based on how citations propagate through the network.
• The algorithm computes a relative score—DataRank—for each dataset, reflecting its inferred influence based on citation flow.
• The approach does not require explicit dataset citations; instead, it infers usage from publication citation patterns.
• The method is validated using metadata from GenBank and Figshare, comparing DataRank scores to actual web visits and download counts.

Experimental results

Research questions

• RQ1 Can a network flow algorithm effectively assign credit to datasets using only publication citation networks?
• RQ2 How do the citation dynamics of datasets differ from those of publications in terms of obsolescence and flow propagation?
• RQ3 To what extent can DataRank predict real-world dataset usage such as web visits and downloads?
• RQ4 How does DataRank compare in predictive accuracy to alternative citation-based ranking methods for datasets?
• RQ5 Can this method support fairer evaluation of scientific impact without requiring changes in citation behavior?

Key findings

• DataRank outperforms alternative methods in predicting actual dataset usage, as measured by web visits to GenBank sequences and downloads of Figshare datasets.
• The algorithm successfully captures the distinct citation dynamics of datasets compared to publications, particularly in terms of obsolescence rates.
• Even with limited citation data—only 693 GenBank records and 355 Figshare datasets found in the network—DataRank achieved meaningful predictive performance.
• The method provides interpretable scores that reflect the relative influence of datasets, offering a foundation for impact assessment in data-intensive science.
• DataRank enables the ranking of datasets without requiring explicit citations, making it a viable tool for attributing credit to digital scientific objects.
• The study demonstrates that datasets can be systematically evaluated for impact using network-based inference, even when citation practices remain incomplete.