[Paper Review] Do We Still Need Clinical Language Models?

Eric Lehman, Evan Hernandez+8 more|arXiv (Cornell University)|Feb 16, 2023

Topic ModelingComputer Science50 citations

TL;DR

The paper compares 12 language models across 3 clinical tasks and finds that small, in-domain trained clinical models outperform in-context learning with large general models, highlighting the value of domain-specific pretraining for clinical text.

ABSTRACT

Although recent advances in scaling large language models (LLMs) have resulted in improvements on many NLP tasks, it remains unclear whether these models trained primarily with general web text are the right tool in highly specialized, safety critical domains such as clinical text. Recent results have suggested that LLMs encode a surprising amount of medical knowledge. This raises an important question regarding the utility of smaller domain-specific language models. With the success of general-domain LLMs, is there still a need for specialized clinical models? To investigate this question, we conduct an extensive empirical analysis of 12 language models, ranging from 220M to 175B parameters, measuring their performance on 3 different clinical tasks that test their ability to parse and reason over electronic health records. As part of our experiments, we train T5-Base and T5-Large models from scratch on clinical notes from MIMIC III and IV to directly investigate the efficiency of clinical tokens. We show that relatively small specialized clinical models substantially outperform all in-context learning approaches, even when finetuned on limited annotated data. Further, we find that pretraining on clinical tokens allows for smaller, more parameter-efficient models that either match or outperform much larger language models trained on general text. We release the code and the models used under the PhysioNet Credentialed Health Data license and data use agreement.

Motivation & Objective

• Assess whether specialized clinical language models are still needed given advances in domain-agnostic LLMs.
• Evaluate a range of models (220M–175B params) on clinically relevant tasks using EHR notes.
• Investigate the effectiveness of pretraining on clinical data vs. general-domain data for model performance and efficiency.
• Analyze when from-scratch clinical pretraining is cost-effective and how in-domain tokens impact performance.

Proposed method

• Train and evaluate 12 language models on three clinical tasks derived from MIMIC notes.
• Develop three Clinical-T5 models (from scratch and with checkpoint initialization) pretrained on MIMIC III/IV.
• Compare specialized clinical models against general-domain models and large LLMs using fine-tuning and in-context learning.
• Use task-specific metrics: MedNLI accuracy, RadQA token-level F1 and exact match, CLIP micro/macro F1.
• Provide FLOPs-based cost analysis to compare pretraining-from-scratch vs. finetuning/inference for different model sizes.

Figure 1 : We consider three options for how a healthcare system with access to clinical notes might approach a clinical problem. First, the healthcare system could use a specialized language model pretrained on clinical notes. This model could be pretrained from scratch (Row 1) or from a publicly a

Experimental results

Research questions

• RQ1 Do small specialized clinical models outperform in-context learning with large general-domain LLMs on clinical note tasks?
• RQ2 Does pretraining on clinical data yield more efficient models (in terms of FLOPs) that match or surpass larger general-domain models?
• RQ3 What is the trade-off between from-scratch clinical pretraining and domain-adaptive pretraining in terms of performance and cost?
• RQ4 How does performance scale with in-domain token usage versus general-domain pretraining across tasks?
• RQ5 Is in-context learning a viable substitute for finetuned clinical models in safety-critical clinical NLP?”

Key findings

• Specialized clinical models as small as 345M parameters substantially outperform in-context learning approaches across tasks.
• Clinical-T5-Large and Clinical-T5-Base show performance gains over their general-domain counterparts, with from-scratch pretraining yielding the largest improvements.
• Clinical models like BioClinRoBERTa and GatorTron achieve the highest task performance, often with far fewer parameters than large general models like T5-XL or GPT-3.
• Pretraining on clinical data enables smaller models to match or exceed larger general-domain models, emphasizing data domain over sheer size.
• In few-shot settings, specialized clinical models generally outperform ICL-based large LLMs, though results vary by task and data availability.
• The study provides a break-even cost analysis indicating that, in many scenarios, training a smaller clinical model from scratch can be more cost-effective than continuously using a larger general-domain model for inference.

Figure 2 : An example of the tasks we consider in this paper. In MedNLI, the goal is determine if the two sentences entail, contradict or are neutral to each other. RadQA is an extractive question answering task over radiology reports. In CLIP, the goal is to identify the different types of patient