[Paper Review] Utilizing ChatGPT to Enhance Clinical Trial Enrollment

Georgios Peikos, Symeon Symeonidis+2 more|arXiv (Cornell University)|Jun 3, 2023

Topic Modeling4 citations

TL;DR

This study proposes using ChatGPT to extract patient-specific information from unstructured clinical notes and generate high-quality search queries for identifying eligible clinical trials. The approach significantly improves retrieval performance over both state-of-the-art methods and human-generated queries, demonstrating ChatGPT's potential to automate and enhance clinical trial enrollment while preserving patient privacy.

ABSTRACT

Clinical trials are a critical component of evaluating the effectiveness of new medical interventions and driving advancements in medical research. Therefore, timely enrollment of patients is crucial to prevent delays or premature termination of trials. In this context, Electronic Health Records (EHRs) have emerged as a valuable tool for identifying and enrolling eligible participants. In this study, we propose an automated approach that leverages ChatGPT, a large language model, to extract patient-related information from unstructured clinical notes and generate search queries for retrieving potentially eligible clinical trials. Our empirical evaluation, conducted on two benchmark retrieval collections, shows improved retrieval performance compared to existing approaches when several general-purposed and task-specific prompts are used. Notably, ChatGPT-generated queries also outperform human-generated queries in terms of retrieval performance. These findings highlight the potential use of ChatGPT to enhance clinical trial enrollment while ensuring the quality of medical service and minimizing direct risks to patients.

Motivation & Objective

• To address the challenge of low clinical trial enrollment rates by automating the identification of eligible trials using patient EHR data.
• To leverage large language models like ChatGPT to extract meaningful, structured information from unstructured clinical notes.
• To improve retrieval performance in clinical trial search by generating more effective, semantically rich queries.
• To evaluate whether LLM-generated queries outperform human-generated and state-of-the-art AI-generated queries.
• To ensure patient privacy and data security during information extraction and query generation.

Proposed method

• Employing the GPT-3.5 model (ChatGPT) to process unstructured clinical notes and extract key patient-related information such as medical conditions, demographics, and family history.
• Designing specialized prompting strategies, including role-based and task-specific prompts (e.g., IEMT), to enhance information extraction and query formulation.
• Transforming extracted patient data into concise, semantically meaningful search queries for clinical trial retrieval systems.
• Using benchmark retrieval collections (e.g., CLEF and TREC) to evaluate query effectiveness via standard retrieval metrics like MAP and P@10.
• Applying retrieval models such as BM25 and RM3 to assess the performance of ChatGPT-generated queries against baselines.
• Implementing privacy-preserving measures, including data anonymization and adherence to HIPAA, to ensure patient confidentiality.

Figure 1: Combining ChatGPT with a negation classification model.

Experimental results

Research questions

• RQ1 Can ChatGPT effectively extract relevant patient information from unstructured clinical notes to support clinical trial search?
• RQ2 Do ChatGPT-generated queries outperform human-generated queries in retrieving relevant clinical trials?
• RQ3 How does the performance of ChatGPT-generated queries compare to state-of-the-art AI-based query generation methods?
• RQ4 What prompting strategies maximize retrieval performance in clinical trial search using LLMs?
• RQ5 What are the risks of hallucination, semantic drift, and misinterpretation (e.g., negation errors) in LLM-generated queries, and how can they be mitigated?

Key findings

• The IEMT prompting strategy—where ChatGPT is assigned a domain-specific role and given a detailed task description—significantly improved retrieval performance on both benchmark collections.
• ChatGPT-generated queries achieved higher mean average precision (MAP) and P@10 scores than state-of-the-art AI-based query generation methods.
• ChatGPT-generated queries outperformed human-generated queries in terms of retrieval effectiveness, demonstrating superior semantic richness and relevance.
• The IEMT and IEMT+RM3 configurations showed statistically significant improvements in retrieval performance, confirming the value of structured prompting.
• ChatGPT demonstrated strong disambiguation capabilities, correctly interpreting complex medical terms and abbreviations in clinical notes.
• Despite strong performance, the model occasionally introduced topical drift when prompted to expand responses, and misinterpreted negated statements, highlighting risks requiring further mitigation.