[Paper Review] Large Language Models with Retrieval-Augmented Generation for Zero-Shot Disease Phenotyping

Will Thompson, David Vidmar+10 more|arXiv (Cornell University)|Dec 11, 2023

Biomedical Text Mining and Ontologies12 citations

TL;DR

The paper presents a zero-shot disease phenotyping method using retrieval-augmented generation with MapReduce to process full EHR notes, applied to pulmonary hypertension, outperforming a physician rules-based phenotype.

ABSTRACT

Identifying disease phenotypes from electronic health records (EHRs) is critical for numerous secondary uses. Manually encoding physician knowledge into rules is particularly challenging for rare diseases due to inadequate EHR coding, necessitating review of clinical notes. Large language models (LLMs) offer promise in text understanding but may not efficiently handle real-world clinical documentation. We propose a zero-shot LLM-based method enriched by retrieval-augmented generation and MapReduce, which pre-identifies disease-related text snippets to be used in parallel as queries for the LLM to establish diagnosis. We show that this method as applied to pulmonary hypertension (PH), a rare disease characterized by elevated arterial pressures in the lungs, significantly outperforms physician logic rules ($F_1$ score of 0.62 vs. 0.75). This method has the potential to enhance rare disease cohort identification, expanding the scope of robust clinical research and care gap identification.

Motivation & Objective

• Motivate scalable, accurate disease phenotyping from broad EHR data without manual rule crafting for rare diseases.
• Introduce a retrieval-augmented generation (RAG) pipeline to pre-identify disease-related snippets from full patient records.
• Evaluate a MapReduce-based aggregation to combine per-snippet inferences into a patient diagnosis.
• Compare LLM-based phenotyping against a physician-developed structured phenotype baseline on unseen data.

Proposed method

• Use regular expressions (Regex) to fetch relevant 2,048-token snippets from patient notes.
• Feed retrieved snippets into a large language model (PaLM-2 variant) for diagnosis in a zero-shot setup.
• Apply MapReduce to query each snippet in parallel and generate per-snippet outputs.
• Compare two aggregation strategies: an LLM-based aggregation and a Max function aggregation.
• Experiment with prompt designs including chain-of-thought and steering to balance history vs. current PH signals.
• Exclude ECHO/CT-related snippets to reduce false positives and improve performance.

Experimental results

Research questions

• RQ1 Can a retrieval-augmented LLM pipeline analyze entire patient records to identify PH without disease-specific rules?
• RQ2 Does MapReduce-based aggregation improve robustness and accuracy of per-snippet inferences into patient-level phenotypes?
• RQ3 How does LLM-based phenotyping compare to traditional SME-developed structured phenotypes for PH in terms of F1 score?
• RQ4 What prompt designs and retrieval exclusions yield the best zero-shot PH diagnosis performance?

Key findings

• LLM-based phenotypes generally outperform the structured phenotype by 18%–21% in F1 on the test set.
• The best-performing configuration (Max aggregation with Regex exclusions) achieved an F1 of 0.73 on the test set.
• Including prompt amendments and ECHO/CT exclusions improved performance on validation, guiding final design choices.
• Retrieved notes spanned 29 distinct note types, highlighting cross-note-type retrieval importance for PH identification.
• There is some drop in F1 from validation to test (0.05–0.10), potentially due to cohort size and overfitting on training data.
• In practice, the LLM-based approach identified roughly twice as many PH-confirmed patients as the structured phenotype.