[Paper Review] WithdrarXiv: A Large-Scale Dataset for Retraction Study

Delip Rao, Jonathan Young+2 more|arXiv (Cornell University)|Dec 4, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper introduces WithdrarXiv, the first large-scale dataset of over 14,000 withdrawn arXiv preprints with associated retraction comments, and develops a 10-category taxonomy of retraction reasons through analysis of author statements. It demonstrates zero-shot LLM-based categorization with a weighted F1-score of 0.9594 and releases WithdrarXiv - SciFy, an enriched version with parsed PDFs for scientific feasibility and verification research, while implementing responsible data sharing practices.

ABSTRACT

Retractions play a vital role in maintaining scientific integrity, yet systematic studies of retractions in computer science and other STEM fields remain scarce. We present WithdrarXiv, the first large-scale dataset of withdrawn papers from arXiv, containing over 14,000 papers and their associated retraction comments spanning the repository's entire history through September 2024. Through careful analysis of author comments, we develop a comprehensive taxonomy of retraction reasons, identifying 10 distinct categories ranging from critical errors to policy violations. We demonstrate a simple yet highly accurate zero-shot automatic categorization of retraction reasons, achieving a weighted average F1-score of 0.96. Additionally, we release WithdrarXiv-SciFy, an enriched version including scripts for parsed full-text PDFs, specifically designed to enable research in scientific feasibility studies, claim verification, and automated theorem proving. These findings provide valuable insights for improving scientific quality control and automated verification systems. Finally, and most importantly, we discuss ethical issues and take a number of steps to implement responsible data release while fostering open science in this area.

Motivation & Objective

• To address the lack of systematic retraction studies in computer science and STEM fields, particularly on preprint repositories like arXiv.
• To create a comprehensive, large-scale dataset of withdrawn arXiv papers and their retraction comments spanning the repository’s entire history through September 2024.
• To develop a robust, human-annotated taxonomy of 10 distinct retraction reasons based on author-provided justifications.
• To evaluate the effectiveness of large language models in zero-shot categorization of retraction reasons.
• To release WithdrarXiv - SciFy, an enriched dataset with parsed full-text PDFs to support research in scientific feasibility, claim verification, and automated theorem proving.

Proposed method

• Harvested all withdrawn arXiv IDs via collaboration with arXiv.org, resulting in 16,460 entries, which were then deduplicated to 14,000 unique withdrawn papers.
• Collected and analyzed retraction comments from authors to develop a 10-category taxonomy of retraction reasons, including factual errors, methodological flaws, and incomplete work.
• Applied zero-shot prompting with large language models to automatically categorize retraction reasons without fine-tuning, leveraging model in-context learning.
• Implemented a rigorous data curation pipeline to ensure consistency and accuracy in the taxonomy, validated through iterative review and refinement.
• Extended the dataset with WithdrarXiv - SciFy, including scripts to parse full-text PDFs for 1,000 selected papers to enable downstream research in scientific feasibility and verification.
• Applied responsible data release practices, including anonymization, access controls, and author privacy protections, informed by principles from recent work on AI ethics and the 'right to be forgotten'.

Experimental results

Research questions

• RQ1 What are the most common reasons for preprint withdrawals on arXiv, and how do they differ from retractions in traditional journal publishing?
• RQ2 To what extent can large language models perform zero-shot categorization of retraction reasons with high accuracy?
• RQ3 How can parsed full-text PDFs from withdrawn preprints be leveraged to improve automated scientific feasibility and claim verification systems?
• RQ4 What ethical and privacy challenges arise in releasing datasets of withdrawn scientific works, and how can they be responsibly mitigated?
• RQ5 How do withdrawal patterns on arXiv compare across scientific disciplines, and what trends emerge over time?

Key findings

• The dataset contains 14,000 unique withdrawn preprints from arXiv’s full history through September 2024, with associated retraction comments.
• A comprehensive 10-category taxonomy of retraction reasons was developed, with factual or methodological errors being the most common (37%) and incomplete work accounting for 19% of withdrawals.
• Zero-shot LLM categorization achieved a weighted average F1-score of 0.9594, demonstrating high accuracy without fine-tuning.
• The most frequent retraction reasons on arXiv differ significantly from biomedical fields, where plagiarism is common; in computer science, errors and incomplete work dominate.
• WithdrarXiv - SciFy was released with parsed full-text PDFs for a subset of papers, enabling future research in automated scientific verification and feasibility checking.
• Responsible data release practices were successfully implemented, including privacy-preserving measures and access controls, to balance utility and ethical concerns.