[Paper Review] ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

Kexin Huang, Jaan Altosaar+1 more|arXiv (Cornell University)|Apr 10, 2019

Machine Learning in HealthcareComputer Science33 references612 citations

TL;DR

ClinicalBERT pre-trains BERT on clinical notes to create a continuous representation of clinical text and fine-tunes it for 30-day hospital readmission prediction, outperforming baselines and enabling interpretability via attention. The model is open-sourced with training/evaluation scripts.

ABSTRACT

Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.

Motivation & Objective

• Demonstrate that clinical notes can be effectively represented with a BERT-based model trained on clinical data.
• Assess ClinicalBERT's performance on predicting 30-day readmission at multiple timepoints during admission.
• Show that ClinicalBERT captures clinically meaningful relationships between medical concepts.
• Provide interpretable predictions via attention mechanisms and release open-source resources for replication.

Proposed method

• Pre-train BERT on clinical notes to create ClinicalBERT with masked language modeling and next sentence prediction objectives.
• Fine-tune ClinicalBERT on the readmission prediction task using the [CLS] representation with a linear classifier.
• Handle long and numerous notes by concatenating sequences and aggregating subsequence predictions (max/mean with a scaling factor).
• Evaluate using clinically relevant metrics: AUROC, AUPRC, and RP80 (recall at 80% precision).
• Compare against Bag-of-Words, bi-LSTM with Word2Vec, and standard BERT pre-trained on non-clinical text; analyze attention for interpretability.
• Open-source the ClinicalBERT parameters and training/evaluation scripts.

Experimental results

Research questions

• RQ1 Can a BERT-based model pre-trained on clinical notes improve 30-day readmission prediction compared to standard language models?
• RQ2 Do clinical notes provide predictive signals for readmission across different admission timepoints (early notes vs discharge summaries)?
• RQ3 Are the attention mechanisms in ClinicalBERT interpretable and indicative of clinically relevant terms?
• RQ4 What is the impact of modeling long clinical notes on predictive performance and interpretability?

Key findings

• ClinicalBERT outperforms baselines in 30-day readmission prediction across discharge summaries and early admission notes.
• On discharge summaries, ClinicalBERT achieves AUROC 0.714±0.018, AUPRC 0.701±0.021, RP80 0.242±0.111, higher than Bag-of-Words, bi-LSTM, and standard BERT baselines.
• ClinicalBERT embeddings correlate with physician judgments of clinical term similarity better than Word2Vec and FastText.
• Attention weights in ClinicalBERT enable interpretation by highlighting terms predictive of readmission (e.g., chronic/acute heart-related terms).
• Model is open-sourced with pre-training and fine-tuning parameters to facilitate replication and adaptation to other clinical tasks.