[Paper Review] GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Records

Xi Yang, Chen, Aokun+16 more|arXiv (Cornell University)|Feb 2, 2022

Topic Modeling52 citations

TL;DR

GatorTron develops a large clinical language model (up to 8.9B parameters) trained on >90B words (including >82B de-identified clinical text) and evaluates it on 5 clinical NLP tasks, showing notable performance gains with scale.

ABSTRACT

There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model - GatorTron - using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve 5 clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.

Motivation & Objective

• Motivate the development of a large-scale clinical language model to better utilize unstructured EHR data.
• Investigate the impact of scaling parameters and data size on clinical NLP task performance.
• Systematically evaluate GatorTron on multiple clinical NLP tasks to assess generalizability across tasks.

Proposed method

• Train GatorTron from scratch with >90B words of text, including >82B de-identified clinical text.
• Scale the model from 110M to 8.9B parameters to study performance gains.
• Evaluate on 5 clinical NLP tasks: clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA).
• Compare performance across model sizes to assess the effect of parameter scaling and data scale.

Experimental results

Research questions

• RQ1 How does increasing model size (parameters) affect performance on clinical NLP tasks?
• RQ2 How does increasing training data size influence results across tasks?
• RQ3 Do large clinical LMs provide consistent gains across clinical NLP tasks such as extraction, relation, similarity, NLI, and QA?
• RQ4 What are the relative improvements in key tasks when scaling both parameters and data?

Key findings

• Model scale from 110M to 8.9B parameters yields improvements across 5 clinical NLP tasks.
• NLI accuracy improves by 9.6% with scaling.
• MQA accuracy improves by 9.5% with scaling.
• Training on >90B words, including >82B de-identified clinical text, supports substantial performance gains.
• GatorTron models are publicly available for use in medical AI systems.