[Paper Review] Automated Clinical Data Extraction with Knowledge Conditioned LLMs

Diya Li, Asim Kadav+3 more|arXiv (Cornell University)|Jun 26, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes a knowledge-conditioned two-stage LLM framework for automated extraction of lung lesion data from clinical reports, using in-context learning (ICL) to align dynamically generated internal knowledge with expert-curated external knowledge. The method improves F1 scores by 12.9% on key fields (size, margin, solidity) compared to existing ICL methods, enhancing accuracy and reducing hallucination in clinical NLP applications.

ABSTRACT

The extraction of lung lesion information from clinical and medical imaging reports is crucial for research on and clinical care of lung-related diseases. Large language models (LLMs) can be effective at interpreting unstructured text in reports, but they often hallucinate due to a lack of domain-specific knowledge, leading to reduced accuracy and posing challenges for use in clinical settings. To address this, we propose a novel framework that aligns generated internal knowledge with external knowledge through in-context learning (ICL). Our framework employs a retriever to identify relevant units of internal or external knowledge and a grader to evaluate the truthfulness and helpfulness of the retrieved internal-knowledge rules, to align and update the knowledge bases. Experiments with expert-curated test datasets demonstrate that this ICL approach can increase the F1 score for key fields (lesion size, margin and solidity) by an average of 12.9% over existing ICL methods.

Motivation & Objective

• To address hallucination and low accuracy in LLM-based clinical data extraction due to lack of domain-specific knowledge.
• To improve the reliability of extracting fine-grained lung lesion features (e.g., margin, solidity) from unstructured radiology reports.
• To develop a scalable, iterative method for aligning self-generated internal knowledge with authoritative external medical knowledge.
• To reduce manual effort in clinical research and radiology workflows by automating structured extraction of lesion findings.

Proposed method

• The framework uses a two-stage process: first detecting lung lesion findings and parsing primary structured fields, then further parsing unstructured lesion description text into detailed fields.
• An internal knowledge base is automatically generated from a curated medical report corpus, transforming relevant references into higher-level extraction rules.
• A retriever selects relevant internal knowledge rules based on input reports, while a grader evaluates their truthfulness and helpfulness using external expert knowledge.
• In-context learning (ICL) prompts the LLM with retrieved and graded rules to improve accuracy and consistency in extraction.
• The system employs a controlled vocabulary in stage two to guide parsing of complex, nested lesion descriptions into structured fields.
• External knowledge is used solely to validate and update the internal knowledge base, enabling dynamic refinement without changing the core architecture.

Figure 1: Example of lung lesion information extraction. Two findings (one describing a single lesion, and the other, two lesions) were identified in the source text. Example rules from the generated internal knowledge base are shown. First-stage finding detection and primary structured field parsin

Experimental results

Research questions

• RQ1 Can in-context learning with dynamically aligned internal and external knowledge reduce hallucination in LLM-based clinical data extraction?
• RQ2 How does knowledge-conditioned prompting improve the accuracy of extracting fine-grained lung lesion features like margin and solidity?
• RQ3 To what extent can a self-updating internal knowledge base outperform static ICL or retrieval-augmented generation (RAG) methods in clinical NLP tasks?
• RQ4 Can a two-stage parsing approach improve the extraction of nested, complex lesion descriptions compared to end-to-end methods?
• RQ5 How does the integration of expert-curated external knowledge enhance the reliability of LLM-generated clinical data extractions?

Key findings

• The proposed framework increased the F1 score for key lesion fields—lesion size, margin, and solidity—by an average of 12.9% compared to existing ICL methods.
• The method demonstrated improved reliability in detecting clinically relevant findings, significantly reducing hallucination in LLM outputs.
• The two-stage parsing approach enabled more accurate extraction of nested lesion description fields by applying specialized prompting and controlled vocabularies.
• The grader module, though currently pre-trained without fine-tuning, effectively evaluated retrieved knowledge rules for truthfulness and helpfulness.
• The system’s architecture allows for easy integration of updated or improved external knowledge sources without modifying the core model.
• The framework outperformed standard RAG and ICL baselines, particularly in extracting complex, domain-specific clinical attributes.

Figure 3: Heatmap of lesion size extraction performance with varying values for the retriever’s top- $k$ hyper-parameter, for both lung-related and lung-irrelevant rules.