[Paper Review] SciDr at SDU-2020: IDEAS -- Identifying and Disambiguating Everyday Acronyms for Scientific Domain

Aadarsh Singh, Priyanshu Kumar|arXiv (Cornell University)|Feb 17, 2021

Biomedical Text Mining and Ontologies25 references4 citations

TL;DR

This paper presents a BERT- and SciBERT-based approach for acronym identification (AI) and disambiguation (AD) in scientific texts, treating AI as sequence tagging and AD as span prediction. The system achieves 11th place in AI and 3rd in AD on the SDU-2020 shared task, leveraging BIOless tagging, model blending, external data, and post-processing to improve performance on low-context acronyms.

ABSTRACT

We present our systems submitted for the shared tasks of Acronym Identification (AI) and Acronym Disambiguation (AD) held under Workshop on SDU. We mainly experiment with BERT and SciBERT. In addition, we assess the effectiveness of "BIOless" tagging and blending along with the prowess of ensembling in AI. For AD, we formulate the problem as a span prediction task, experiment with different training techniques and also leverage the use of external data. Our systems rank 11th and 3rd in AI and AD tasks respectively.

Motivation & Objective

• Address the challenge of identifying and disambiguating acronyms in scientific documents, where manual annotation is costly and time-consuming.
• Develop automated NLP systems that mimic domain experts in detecting acronyms and selecting their correct long-form expansions.
• Improve performance on acronym identification and disambiguation by leveraging domain-specific pretraining (SciBERT) and advanced deep learning techniques.
• Investigate the impact of data augmentation, external knowledge, and model ensembling on model robustness and accuracy in low-resource contexts.
• Identify limitations in model predictions and dataset annotation, particularly for acronyms with insufficient contextual cues.

Proposed method

• Treat acronym identification as a sequence tagging problem using BERT and SciBERT, with BIOless tagging to simplify labeling and improve training efficiency.
• Apply model blending and hard voting ensembling strategies to combine predictions from multiple models, enhancing robustness and generalization.
• Formulate acronym disambiguation as a span prediction task, where the model selects the correct long-form expansion from a list of candidates based on context.
• Fine-tune SciBERT on the SDU-2020 dataset using two-stage training and incorporate external scientific text to improve knowledge infusion and performance on rare acronyms.
• Implement a post-processing rule to directly predict the long form when the acronym is parenthetically expanded in the text, correcting model errors in such cases.
• Evaluate model performance using cross-validation and test set predictions, analyzing discrepancies between individual models and their ensemble to guide improvements.

Experimental results

Research questions

• RQ1 How effective is SciBERT compared to general BERT for acronym identification and disambiguation in scientific texts?
• RQ2 To what extent does BIOless tagging improve model performance in acronym identification compared to standard BIO tagging?
• RQ3 Can model ensembling and blending significantly improve disambiguation accuracy, especially for ambiguous or low-frequency acronyms?
• RQ4 How does incorporating external scientific text enhance model generalization and performance on rare or out-of-vocabulary acronyms?
• RQ5 What are the main failure modes of the models, and how do dataset annotation issues (e.g., insufficient context) affect prediction reliability?

Key findings

• The system achieved 11th place in the Acronym Identification (AI) task and 3rd place in the Acronym Disambiguation (AD) task at the SDU-2020 shared evaluation.
• SciBERT outperformed general BERT due to its pretraining on scientific corpora, demonstrating the benefit of domain-specific language modeling.
• BIOless tagging and model blending significantly improved AI performance, particularly in handling complex or ambiguous acronym-long form pairs.
• Ensembling reduced prediction variance and improved accuracy on 92 test instances where individual models disagreed, especially for acronyms with limited context.
• Post-processing based on parenthetical expansions corrected model errors in 15% of cases where the long form was explicitly written in parentheses.
• The models struggled most with acronyms that lacked sufficient context or had multiple plausible expansions (e.g., LPP, GCN), indicating a need for better context modeling or data augmentation.