[Paper Review] Librarian-in-the-Loop: A Natural Language Processing Paradigm for Detecting Informal Mentions of Research Data in Academic Literature

Lizhou Fan, Sara Lafia+4 more|arXiv (Cornell University)|Mar 10, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a librarian-in-the-loop natural language processing (NLP) paradigm to detect informal mentions of research datasets in academic literature, combining iterative human annotation with machine learning to improve Named Entity Recognition (NER) for dataset names. The approach leverages ICPSR librarians' expertise to generate high-quality training data, resulting in a robust, domain-specific NLP model that enhances data citation discovery and supports comprehensive research data impact metrics.

ABSTRACT

Data citations provide a foundation for studying research data impact. Collecting and managing data citations is a new frontier in archival science and scholarly communication. However, the discovery and curation of research data citations is labor intensive. Data citations that reference unique identifiers (i.e. DOIs) are readily findable; however, informal mentions made to research data are more challenging to infer. We propose a natural language processing (NLP) paradigm to support the human task of identifying informal mentions made to research datasets. The work of discovering informal data mentions is currently performed by librarians and their staff in the Inter-university Consortium for Political and Social Research (ICPSR), a large social science data archive that maintains a large bibliography of data-related literature. The NLP model is bootstrapped from data citations actively collected by librarians at ICPSR. The model combines pattern matching with multiple iterations of human annotations to learn additional rules for detecting informal data mentions. These examples are then used to train an NLP pipeline. The librarian-in-the-loop paradigm is centered in the data work performed by ICPSR librarians, supporting broader efforts to build a more comprehensive bibliography of data-related literature that reflects the scholarly communities of research data users.

Motivation & Objective

• Address the challenge of identifying informal mentions of research datasets in academic literature, which are often overlooked in formal citation systems.
• Reduce the labor-intensive burden of manual curation for data citation bibliographies by automating detection of informal dataset references.
• Improve the comprehensiveness of the ICPSR Bibliography of Data-related Literature by capturing mentions beyond formal data citations with persistent identifiers.
• Develop a scalable, human-in-the-loop NLP pipeline that integrates domain expertise with machine learning for better performance in detecting dataset mentions.
• Support the creation of more accurate research data impact metrics by enabling broader discovery of data reuse across scholarly literature.

Proposed method

• Use a three-level sentence extraction method (HIGH, MID, LOW) based on regular expressions and keyword patterns to identify candidate sentences containing potential dataset mentions.
• Apply iterative NER annotation using Prodigy, with three annotation modes: ner.manual (initial labeling), ner.correct (error correction), and ner.teach (binary acceptance/rejection of model predictions).
• Leverage librarian expertise to iteratively refine NER labels (DATASET spans) and improve rule-based patterns, enhancing model generalization and accuracy.
• Train a machine learning-based NER model on cumulative, human-annotated data, using the librarian’s domain knowledge to bootstrap and refine the model in a feedback loop.
• Integrate pattern-based detection (e.g., 'American National Election Survey', 'ANES') with learned NER to improve precision in identifying dataset names and acronyms.
• Use the resulting high-quality, domain-specific dataset as a benchmark for future NLP research in scholarly communication and science of science studies.

Experimental results

Research questions

• RQ1 How can informal mentions of research datasets in academic literature be systematically detected when they lack persistent identifiers?
• RQ2 To what extent can librarian expertise, when integrated iteratively into NLP model training, improve the detection of dataset mentions compared to purely automated methods?
• RQ3 What is the impact of combining rule-based pattern matching with human-in-the-loop NER annotation on model performance and annotation efficiency?
• RQ4 Can a librarian-in-the-loop workflow produce a high-quality, reusable NLP training dataset for detecting dataset mentions in scholarly texts?
• RQ5 How does this approach support the development of more comprehensive research data impact metrics across social science research communities?

Key findings

• The librarian-in-the-loop NLP paradigm significantly improves the detection of informal dataset mentions by combining human expertise with iterative model training.
• The use of iterative annotation (ner.manual, ner.correct, ner.teach) in Prodigy enables efficient and accurate labeling of dataset name spans, even in complex or ambiguous contexts.
• The model achieves high precision in identifying dataset mentions, allowing librarians to focus their curation efforts on high-confidence candidates rather than exhaustive manual screening.
• The resulting NER model and annotated dataset serve as a high-quality benchmark for future research in computational social science and scholarly communication.
• The approach enables the expansion of the ICPSR Bibliography to include a broader range of data reuse references, enhancing the comprehensiveness of data impact metrics.
• The workflow is scalable and adaptable, offering a sustainable model for data curation in institutions with limited resources, particularly in social science data archives.