[Paper Review] 'I Updated the ': The Evolution of References in the English Wikipedia and the Implications for Altmetrics

Olga Zagovora, Roberto Ulloa+2 more|arXiv (Cornell University)|Oct 6, 2020

Wikis in Education and Collaboration17 references4 citations

TL;DR

This paper introduces a publicly available dataset of over 55 million Wikipedia references from the English Wikipedia up to June 2019, using a novel method to track reference creation, modifications, deletions, and reinsertions with high accuracy. The study reveals a sustained, human-driven effort to improve reference quality through increasing use of persistent identifiers (e.g., DOI, ISBN), highlighting the need to incorporate reference evolution into altmetrics design.

ABSTRACT

With this work, we present a publicly available dataset of the history of all the references (more than 55 million) ever used in the English Wikipedia until June 2019. We have applied a new method for identifying and monitoring references in Wikipedia, so that for each reference we can provide data about associated actions: creation, modifications, deletions, and reinsertions. The high accuracy of this method and the resulting dataset was confirmed via a comprehensive crowdworker labelling campaign. We use the dataset to study the temporal evolution of Wikipedia references as well as users' editing behaviour. We find evidence of a mostly productive and continuous effort to improve the quality of references: (1) there is a persistent increase of reference and document identifiers (DOI, PubMedID, PMC, ISBN, ISSN, ArXiv ID), and (2) most of the reference curation work is done by registered humans (not bots or anonymous editors). We conclude that the evolution of Wikipedia references, including the dynamics of the community processes that tend to them should be leveraged in the design of relevance indexes for altmetrics, and our dataset can be pivotal for such effort.

Motivation & Objective

• To create a comprehensive, publicly accessible dataset chronicling the full history of references in the English Wikipedia.
• To develop and validate a method for accurately identifying and tracking reference edits, including creation, modification, deletion, and reinsertion.
• To analyze the temporal dynamics of reference curation and assess the role of registered users versus bots and anonymous editors.
• To evaluate the implications of reference evolution for the design of relevance indexes in altmetrics systems.
• To provide empirical evidence on the quality and sustainability of Wikipedia's reference curation processes.

Proposed method

• A novel algorithmic approach was developed to parse and track all reference edits in Wikipedia's revision history, identifying reference-related actions with high precision.
• The method leverages structured metadata and text patterns to distinguish reference edits from general article edits, enabling fine-grained logging of reference lifecycle events.
• A crowdworker labeling campaign was conducted to validate the accuracy of the method, ensuring reliable identification of reference actions.
• The dataset was constructed by processing all Wikipedia revisions up to June 2019, capturing every reference change with associated timestamps and user identities.
• Persistent identifiers (DOI, PubMedID, ISBN, etc.) were extracted and tracked over time to assess trends in reference quality and standardization.
• Statistical and temporal analysis was applied to study editing patterns, user contributions, and the evolution of reference types.

Experimental results

Research questions

• RQ1 How has the use of persistent identifiers (e.g., DOI, ISBN) in Wikipedia references evolved over time?
• RQ2 What is the distribution of reference curation work between registered users, bots, and anonymous editors?
• RQ3 To what extent do reference edits reflect a continuous, quality-oriented improvement process?
• RQ4 How do reference modifications, deletions, and reinsertions correlate with article content changes?
• RQ5 What role does the reference evolution process play in shaping the relevance of Wikipedia articles for altmetrics?

Key findings

• There is a persistent and significant increase in the use of persistent identifiers such as DOI, PubMedID, and ISBN in Wikipedia references over time.
• The majority of reference curation work is performed by registered human editors, not bots or anonymous users, indicating a strong community-driven quality effort.
• The dataset achieves high accuracy in tracking reference actions, validated through a comprehensive crowdworker labeling campaign.
• Reference modifications, deletions, and reinsertions are common and often part of a sustained effort to improve source reliability and accuracy.
• The temporal dynamics of reference editing suggest an ongoing, systematic process of quality assurance in Wikipedia's content.
• The findings support the integration of reference evolution into altmetrics systems to better reflect article relevance and scholarly impact.