[Paper Review] Zero-shot information extraction from radiological reports using ChatGPT

Danqing Hu, Bing Liu+3 more|arXiv (Cornell University)|Sep 4, 2023

Topic ModelingComputer Science3 citations

TL;DR

This study proposes a zero-shot information extraction framework using ChatGPT to extract structured clinical information from radiological reports without annotated data or model fine-tuning. By designing prompt templates and applying post-processing, ChatGPT achieves competitive performance on simple IE tasks like tumor location and size, though struggles with complex queries such as tumor density and lymph node status due to reasoning and consistency limitations.

ABSTRACT

Electronic health records contain an enormous amount of valuable information, but many are recorded in free text. Information extraction is the strategy to transform the sequence of characters into structured data, which can be employed for secondary analysis. However, the traditional information extraction components, such as named entity recognition and relation extraction, require annotated data to optimize the model parameters, which has become one of the major bottlenecks in building information extraction systems. With the large language models achieving good performances on various downstream NLP tasks without parameter tuning, it becomes possible to use large language models for zero-shot information extraction. In this study, we aim to explore whether the most popular large language model, ChatGPT, can extract useful information from the radiological reports. We first design the prompt template for the interested information in the CT reports. Then, we generate the prompts by combining the prompt template with the CT reports as the inputs of ChatGPT to obtain the responses. A post-processing module is developed to transform the responses into structured extraction results. We conducted the experiments with 847 CT reports collected from Peking University Cancer Hospital. The experimental results indicate that ChatGPT can achieve competitive performances for some extraction tasks compared with the baseline information extraction system, but some limitations need to be further improved.

Motivation & Objective

• To investigate whether large language models like ChatGPT can perform zero-shot information extraction from radiological reports without annotated data or fine-tuning.
• To address the bottleneck of data annotation in traditional information extraction systems by leveraging the zero-shot capabilities of LLMs.
• To design effective prompt templates that guide ChatGPT to extract structured clinical entities from free-text CT reports.
• To evaluate the performance of ChatGPT against a baseline IE system on real-world lung cancer CT reports.
• To identify limitations in ChatGPT’s reasoning, consistency, and privacy handling for clinical deployment.

Proposed method

• Designing task-specific prompt templates for key radiological entities such as tumor location, size, spiculation, and lymph node status.
• Generating input prompts by combining the template with actual CT reports to guide ChatGPT’s response generation.
• Applying a post-processing module to convert model outputs into standardized, structured extraction results.
• Using 847 real CT reports from Peking University Cancer Hospital for evaluation.
• Employing inter-annotator agreement and consistency analysis to assess reliability of ChatGPT’s outputs.
• Incorporating additional instructions in prompts to improve accuracy on ambiguous or complex queries.

Experimental results

Research questions

• RQ1 Can ChatGPT perform zero-shot information extraction from radiological reports with high accuracy and reliability?
• RQ2 How does ChatGPT’s performance compare to a traditional machine learning-based IE system on clinically relevant entities?
• RQ3 What are the main failure modes of ChatGPT in extracting complex radiological concepts like tumor density and lymph node status?
• RQ4 How consistent are ChatGPT’s outputs across multiple inference runs for the same input?
• RQ5 Can prompt engineering improve ChatGPT’s accuracy and reduce hallucination in clinical information extraction?

Key findings

• ChatGPT achieved competitive performance on simple IE tasks such as tumor location, long and short diameters, and pleural invasion, with high consistency and low error rates.
• For complex queries like tumor density and lymph node status, ChatGPT showed significant limitations, often producing inconsistent or incorrect responses despite added prompt instructions.
• ChatGPT frequently misclassified similar terms—e.g., 'streaky' as 'spiculation' or 'pleural thickening' as 'pleural invasion'—indicating challenges in semantic discrimination.
• The model exhibited variable outputs across multiple runs for the same prompt, raising concerns about reliability and reproducibility.
• ChatGPT struggled to recognize synonymous clinical terms (e.g., 'interlobar fissure' and 'interlobar pleura'), highlighting reasoning gaps in medical terminology.
• The study identified privacy risks due to sending raw clinical data to external LLMs, suggesting the need for on-premise deployment of smaller, localized LLMs.