[Paper Review] Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets

Yifan Peng, Shankai Yan+1 more|arXiv (Cornell University)|Jun 13, 2019

Topic ModelingComputer Science39 references70 citations

TL;DR

The BLUE benchmark evaluates BERT and ELMo on ten biomedical/clinical datasets across five tasks and finds that BERT models pre-trained on PubMed abstracts and MIMIC-III clinical notes yield the strongest overall performance, highlighting the value of cross-genre pre-training in biomedicine.

ABSTRACT

Inspired by the success of the General Language Understanding Evaluation benchmark, we introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark to facilitate research in the development of pre-training language representations in the biomedicine domain. The benchmark consists of five tasks with ten datasets that cover both biomedical and clinical texts with different dataset sizes and difficulties. We also evaluate several baselines based on BERT and ELMo and find that the BERT model pre-trained on PubMed abstracts and MIMIC-III clinical notes achieves the best results. We make the datasets, pre-trained models, and codes publicly available at https://github.com/ncbi-nlp/BLUE_Benchmark.

Motivation & Objective

• Introduce the Biomedical Language Understanding Evaluation (BLUE) benchmark with five tasks across biomedical and clinical domains.
• Assess baseline models based on BERT and ELMo to establish strong biomedicine language representations.
• Demonstrate the impact of pre-training across text genres on downstream biomedical NLP tasks.

Proposed method

• Construct BLUE: five tasks with ten corpora spanning biomedical literature and clinical notes.
• Pre-train BERT on PubMed abstracts and MIMIC-III clinical notes (and a PubMed+MIMIC-III combination).
• Fine-tune BERT models on each BLUE task with task-specific input formatting (e.g., sentence pairs for similarity, BIO tagging for NER).
• Compare against ELMo-based baselines pre-trained on PubMed abstracts.
• Provide publicly available datasets, pretrained models, and code for fair benchmarking.

Experimental results

Research questions

• RQ1 How do BERT- and ELMO-based representations perform across a diverse set of biomedical and clinical NLP tasks?
• RQ2 Does pre-training on multiple text genres (biomedical and clinical) improve performance compared with single-genre pre-training?
• RQ3 Which task types (NER, relation extraction, sentence similarity, document classification, inference) benefit most from contextualized language models in biomedicine?

Key findings

• BERT models pre-trained on PubMed abstracts and MIMIC-III achieve the best results across the BLUE tasks.
• BERT with PubMed+MIMIC-III pre-training generally outperforms single-genre pre-training, underscoring cross-genre transfer learning benefits.
• The Base-size BERT models often outperform Large-size models on several tasks, likely due to data size and sentence length factors.
• ELMo baselines lag behind the best BERT configurations on most tasks, highlighting the advantage of modern contextualized transformers in biomedicine.
• On sentence similarity, document classification, and some NER tasks, BERT-Base(P+M) or BERT-Large(P) show strong performance variations tied to dataset characteristics (e.g., average sentence length).
• The BLUE benchmark provides extensive code and pretrained models to facilitate fair comparisons in biomedical NLP.