[Paper Review] Improving astroBERT using Semantic Textual Similarity

Félix Grèzes, Thomas J. Allen+14 more|arXiv (Cornell University)|Nov 29, 2022

Data Quality and Management4 citations

TL;DR

This paper introduces astroBERT, a fine-tuned BERT-based language model pre-trained on 400K astrophysics papers from the NASA ADS database, achieving superior performance on named entity recognition in astronomy literature. It outperforms BERT and SciBERT on the DEAL benchmark, with a test F1 score of 0.6362, and proposes enhancing it using citation context and semantic textual similarity for improved scientific text understanding.

ABSTRACT

The NASA Astrophysics Data System (ADS) is an essential tool for researchers that allows them to explore the astronomy and astrophysics scientific literature, but it has yet to exploit recent advances in natural language processing. At ADASS 2021, we introduced astroBERT, a machine learning language model tailored to the text used in astronomy papers in ADS. In this work we: - announce the first public release of the astroBERT language model; - show how astroBERT improves over existing public language models on astrophysics specific tasks; - and detail how ADS plans to harness the unique structure of scientific papers, the citation graph and citation context, to further improve astroBERT.

Motivation & Objective

• Address the challenge of ambiguous terminology and entity discovery in astronomy literature, such as distinguishing the 'Planck mission' from the 'Planck constant'.
• Improve named entity recognition (NER) in astrophysics texts by creating a domain-specific language model tailored to scientific terminology and paper structure.
• Release a publicly accessible version of astroBERT to accelerate research in astronomy NLP and enable fine-tuning for custom tasks.
• Leverage unique structural features of scientific papers—particularly the citation graph and citation context—to further enhance the model's semantic understanding.
• Explore the use of semantic textual similarity (STS) via sentence embeddings from cited paper abstracts and citing context to improve representation learning beyond standard pretraining tasks.

Proposed method

• Pretrain a base astroBERT model on ~400K recent astrophysics papers (~4B tokens) using Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) over 40 epochs.
• Fine-tune astroBERT on the DEAL benchmark dataset, which contains 32 types of astrophysics-relevant named entities in full-text and acknowledgment sections.
• Release the model, tokenizer, and tutorials via Hugging Face to enable community access and integration into downstream NLP tasks.
• Propose extending astroBERT's pretraining by incorporating semantically similar sentence pairs extracted from citation contexts—specifically, the paragraph preceding a citation and the abstract of the cited paper.
• Utilize Sentence-BERT-style training to learn dense sentence embeddings that reflect semantic similarity, improving contextual understanding in scientific text.
• Replace or augment traditional NSP with citation-based semantic similarity learning to better align with the logical and structural flow of scientific literature.

Experimental results

Research questions

• RQ1 Can a domain-specific language model like astroBERT outperform general-purpose models such as BERT and SciBERT on named entity recognition tasks in astrophysics literature?
• RQ2 How does pretraining on a large, specialized corpus of astrophysics papers affect the model's ability to disambiguate scientific entities and recognize rare or ambiguous terms?
• RQ3 To what extent can citation context and semantic textual similarity improve the performance and robustness of a scientific language model beyond standard pretraining objectives?
• RQ4 What are the key limitations of current astroBERT versions, and which entity types (e.g., Event, Identifier) show the most room for improvement based on precision and recall analysis?
• RQ5 How can the unique structure of scientific papers—especially the citation graph—be systematically leveraged to enhance representation learning in scientific NLP?

Key findings

• astroBERT achieves a test F1 score of 0.6362 on the DEAL benchmark, significantly outperforming BERT (0.4409) and SciBERT (0.5398), demonstrating its superiority in domain-specific NER.
• The public release version of astroBERT achieves a test F1 score of 0.6362, with a Matthews correlation coefficient (MCC) of 0.8302, indicating strong balanced performance across all classes.
• astroBERT shows particular strength in identifying Observational Techniques (ObT) and Databases (DaB), with high precision and recall, suggesting effective learning of technical terminology.
• The model performs less effectively on rare or ambiguous labels such as Event (Eve) and Identifier (Ide), indicating a need for more training data or improved data augmentation strategies.
• The use of citation context for training—pairing citing paragraphs with cited paper abstracts—provides a promising path to improve semantic understanding beyond standard pretraining objectives.
• The model's performance on the validation and test splits is consistently higher than that of the baseline astroBERT version used in the DEAL shared task, confirming the effectiveness of the final training and fine-tuning pipeline.