[Paper Review] SciBERT: A Pretrained Language Model for Scientific Text

Iz Beltagy, Kyle Lo+1 more|arXiv (Cornell University)|Mar 26, 2019

Topic Modeling27 references48 citations

TL;DR

SciBert improves over BERT-Base by pretraining on a large scientific corpus with an in-domain vocabulary, achieving new SOTA results on multiple scientific NLP tasks. Finetuning SciBert generally yields better results than using frozen embeddings.

ABSTRACT

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.

Motivation & Objective

• Motivate the need for domain-specific pretraining due to limited high-quality labeled scientific data.
• Propose SciBert as a Bert-based model trained on a large scientific corpus.
• Evaluate finetuning versus frozen embeddings and the impact of an in-domain vocabulary.
• Show that SciBert achieves improvements and new SOTA on diverse scientific NLP tasks.

Proposed method

• Adopt Bert architecture with two-token prediction objectives for pretraining on scientific text.
• Build SciVocab with SentencePiece to create a 30K in-domain vocabulary.
• Train SciBert on 1.14M Semantic Scholar papers (3.17B tokens) with full text.
• Evaluate both cased/uncased and BaseVocab/SciVocab variants across multiple tasks.
• Fine-tune or freeze embeddings and compare architectures to assess benefits of in-domain pretraining.
• Use dropout, Adam optimization, and slanted triangular learning rate schedule; evaluate with standard metrics.

Experimental results

Research questions

• RQ1 Does in-domain pretraining on scientific text improve performance over generic BERT on scientific NLP tasks?
• RQ2 What is the impact of using an in-domain SciVocab versus the original BaseVocab on downstream tasks?
• RQ3 Are task-specific architectures atop frozen embeddings competitive with full fine-tuning of SciBert?
• RQ4 How does SciBert perform across biomedical, computer science, and multidomain datasets compared to prior SOTA?

Key findings

• SciBert outperforms BERT-Base on scientific tasks by +2.11 F1 with finetuning and +2.43 F1 without (averaged across datasets).
• SciBert achieves new state-of-the-art results on several tasks, especially in biomedical and computer science domains.
• Compared with BioBert and other domain models, SciBert shows competitive or superior performance on multiple datasets such as BC5CDR, ChemProt, ACL-ARC, and SciCite.
• Final averages show SciBert finetuned results surpass the baselines more than frozen embeddings in most domains; finetuning provides the largest gains.
• Using an in-domain SciVocab yields modest additional gains (+0.60 F1 on average) over BaseVocab, indicating corpus pretraining drives most benefits.