[Paper Review] Improving Patient Pre-screening for Clinical Trials: Assisting Physicians with Large Language Models

Danny M. den Hamer, Perry Schoor+2 more|arXiv (Cornell University)|Apr 14, 2023

Artificial Intelligence in Healthcare and Education13 citations

TL;DR

The study uses InstructGPT with a hybrid prompting strategy to assist physicians in pre-screening patients for clinical trials, achieving high recall with a physician-in-the-loop and substantial reduction in criteria to review.

ABSTRACT

Physicians considering clinical trials for their patients are met with the laborious process of checking many text based eligibility criteria. Large Language Models (LLMs) have shown to perform well for clinical information extraction and clinical reasoning, including medical tests, but not yet in real-world scenarios. This paper investigates the use of InstructGPT to assist physicians in determining eligibility for clinical trials based on a patient's summarised medical profile. Using a prompting strategy combining one-shot, selection-inference and chain-of-thought techniques, we investigate the performance of LLMs on 10 synthetically created patient profiles. Performance is evaluated at four levels: ability to identify screenable eligibility criteria from a trial given a medical profile; ability to classify for each individual criterion whether the patient qualifies; the overall classification whether a patient is eligible for a clinical trial and the percentage of criteria to be screened by physician. We evaluated against 146 clinical trials and a total of 4,135 eligibility criteria. The LLM was able to correctly identify the screenability of 72% (2,994/4,135) of the criteria. Additionally, 72% (341/471) of the screenable criteria were evaluated correctly. The resulting trial level classification as eligible or ineligible resulted in a recall of 0.5. By leveraging LLMs with a physician-in-the-loop, a recall of 1.0 and precision of 0.71 on clinical trial level can be achieved while reducing the amount of criteria to be checked by an estimated 90%. LLMs can be used to assist physicians with pre-screening of patients for clinical trials. By forcing instruction-tuned LLMs to produce chain-of-thought responses, the reasoning can be made transparent to and the decision process becomes amenable by physicians, thereby making such a system feasible for use in real-world scenarios.

Motivation & Objective

• Motivate reducing physician time spent on eligibility screening for clinical trials.
• Investigate whether LLMs can identify screenable eligibility criteria from trial text given a patient profile.
• Assess per-criterion and per-trial classification performance of LLM-based screening.
• Evaluate the impact of physician-in-the-loop on recall, precision, and workload reduction.

Proposed method

• Use InstructGPT with prompting that combines one-shot, selection-inference, and chain-of-thought techniques.
• Test on 10 synthetically created patient profiles.
• Evaluate across 146 clinical trials and 4,135 eligibility criteria.
• Measure criteria screenability identification, per-criterion qualification, overall trial eligibility, and percentage of criteria requiring physician review.

Experimental results

Research questions

• RQ1 Can an LLM identify which eligibility criteria are screenable for a given patient profile?
• RQ2 Can an LLM classify whether a patient qualifies for each screenable criterion?
• RQ3 What is the overall eligibility classification (eligible vs ineligible) for a trial given a patient profile?
• RQ4 How much of the trial criteria can be redirected from physician to the LLM (i.e., reduction in criteria to be checked) while maintaining useful recall/precision?
• RQ5 Does forcing chain-of-thought reasoning improve transparency and real-world usability for physicians?

Key findings

• The LLM correctly identified the screenability of 72% of criteria (2,994/4,135).
• The LLM correctly evaluated 72% of the screenable criteria (341/471).
• Trial-level eligibility recall was 0.5 without physician-in-the-loop.
• With a physician-in-the-loop, recall reached 1.0 and precision 0.71 for trial-level eligibility, while reducing the number of criteria to be checked by about 90%.
• Forcing chain-of-thought responses made the reasoning transparent, aiding real-world feasibility.