[Paper Review] Publicly Available Clinical BERT Embeddings

Emily Alsentzer, John R. Murphy+5 more|arXiv (Cornell University)|Apr 6, 2019

Topic ModelingComputer Science20 references720 citations

TL;DR

The paper pretrains and publicly releases clinical-domain BERT models (Clinical BERT and Discharge Summary BERT) trained on MIMIC notes, showing improvements on MedNLI and i2b2 NER tasks over general BERT and BioBERT, but not on de-identification tasks.

ABSTRACT

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Motivation & Objective

• Motivate the need for domain-specific contextual embeddings in clinical text due to linguistic differences from general and biomedical text.
• Pretrain and publicly release Clinical BERT models on MIMIC notes, including a discharge-summaries variant.
• Evaluate clinical BERT models on standard clinical NLP tasks to assess gains over general BERT and BioBERT.

Proposed method

• Train two BERT variants on MIMIC clinical text: Clinical BERT (all notes) and Discharge Summary BERT (discharge summaries).
• Fine-tune the pre-trained models on downstream tasks with a single linear classifier atop BERT outputs.
• Evaluate on MedNLI and four i2b2 NER tasks, plus two de-identification tasks, comparing to BERT and BioBERT baselines.
• Use standard BERT training/setup, with pretraining details provided in appendices (e.g., sequence length, steps).
• Report both quantitative metrics (accuracy, exact F1) and qualitative embedding analysis (nearest neighbors).

Experimental results

Research questions

• RQ1 Do clinically trained BERT models improve performance on clinical NLP tasks compared with general-domain BERT and BioBERT?
• RQ2 Does note-type specific training (all notes vs. discharge summaries) yield task-specific gains?
• RQ3 Are clinical BERT embeddings effective for non-de-identification tasks and less so for de-identification tasks due to data distribution differences?
• RQ4 What qualitative differences emerge between Clinical BERT and BioBERT in clinical contexts?

Key findings

• Clinical BERT improves over general BERT and BioBERT on several clinical tasks, achieving new state-of-the-art on MedNLI (82.7% accuracy).
• On i2b2 2010 and 2012 tasks, Clinical BERT performs well but not always better than BioBERT; Bio+Clinical BERT often yields top results.
• Discharge Summary BERT and note-type specific training can offer additional gains on some tasks.
• Clinical BERT shows no improvements on the two de-identification tasks, likely due to domain shift between de-identification data and the MIMIC-derived training corpus.
• Qualitative analysis indicates Clinical BERT forms more cohesive clinical term associations than BioBERT.