[Paper Review] Large Language Models are Few-Shot Clinical Information Extractors

Monica Agrawal, Stefan Hegselmann+3 more|arXiv (Cornell University)|May 25, 2022

Topic Modeling48 citations

TL;DR

The paper shows that GPT-3-style large language models can perform zero- and few-shot clinical information extraction across diverse structured-output tasks, using guided prompts and lightweight resolvers, and demonstrates training smaller models via weak supervision.

ABSTRACT

A long-running goal of the clinical NLP community is the extraction of important variables trapped in clinical notes. However, roadblocks have included dataset shift from the general domain and a lack of public clinical corpora and annotations. In this work, we show that large language models, such as InstructGPT, perform well at zero- and few-shot information extraction from clinical text despite not being trained specifically for the clinical domain. Whereas text classification and generation performance have already been studied extensively in such models, here we additionally demonstrate how to leverage them to tackle a diverse set of NLP tasks which require more structured outputs, including span identification, token-level sequence classification, and relation extraction. Further, due to the dearth of available data to evaluate these systems, we introduce new datasets for benchmarking few-shot clinical information extraction based on a manual re-annotation of the CASI dataset for new tasks. On the clinical extraction tasks we studied, the GPT-3 systems significantly outperform existing zero- and few-shot baselines.

Motivation & Objective

• Demonstrate that large language models (LLMs) can perform zero- and few-shot clinical information extraction without domain-specific training.
• Introduce new annotated datasets for few-shot clinical IE tasks derived from CASI.
• Show how guided prompts and lightweight resolvers enable structured outputs (span identification, token-level classification, relation extraction).
• Evaluate GPT-3-style systems against zero-shot, few-shot, and supervised baselines across multiple clinical IE tasks.
• Illustrate how weak supervision from LLM outputs can train smaller, deployable models.

Proposed method

• Design guided prompt templates with one-shot examples to steer LLM outputs into structured formats.
• Define a resolver R that maps the LLM string outputs to task-specific structured label spaces (e.g., lists, spans, relations).
• Annotate new datasets by re-annotating CASI snippets for five clinical IE tasks (sense, evidence extraction, coreference, medication status, medication attributes).
• Benchmark GPT-3-like systems against zero-shot and few-shot baselines, plus distilled models trained with weak supervision.
• Compare token-level, phrase-level, and end-to-end relation extraction formulations across tasks.

Experimental results

Research questions

• RQ1 Can GPT-3-style LLMs achieve competitive performance on diverse clinical IE tasks with zero- or few-shot prompts?
• RQ2 Do guided prompts and resolvers reduce post-processing and improve structured outputs in clinical IE?
• RQ3 How well can weak supervision from LLM outputs train smaller, deployable models for clinical IE?
• RQ4 What are the strengths and limitations of LLM-based approaches for clinical sense disambiguation, evidence extraction, coreference, and medication extraction?

Key findings

• GPT-3 + R (with guided prompts) outperforms zero-shot baselines on CASI and MIMIC reverse-substitution for clinical sense disambiguation.
• Weak supervision with PubMedBERT further boosts CASI performance and enables competitive transfer to MIMIC data.
• For biomedical evidence extraction, GPT-3 + R yields higher token-level F1 and arm-identification accuracy than several supervised baselines in the evaluated setup.
• Coreference resolution with GPT-3 + R (1-shot) achieves competitive recall/precision against non-clinical long-document baselines and outperforms them under certain configurations.
• Medication extraction tasks show that 1-shot GPT-3 + R achieves higher recall and precision than ScispaCy baselines across token, phrase, and relation extraction framings.
• Across tasks, guided prompts reduce resolver complexity (often <10 lines of code) and improve structured outputs.