[论文解读] Improving Patient Pre-screening for Clinical Trials: Assisting Physicians with Large Language Models

Danny M. den Hamer, Perry Schoor+2 more|arXiv (Cornell University)|Apr 14, 2023

Artificial Intelligence in Healthcare and Education被引用 13

一句话总结

本研究使用 InstructGPT 结合混合提示策略，协助医生对临床试验的患者进行预筛选，在医生在环的情况下实现高召回率，并显著降低需要审查的标准。

ABSTRACT

Physicians considering clinical trials for their patients are met with the laborious process of checking many text based eligibility criteria. Large Language Models (LLMs) have shown to perform well for clinical information extraction and clinical reasoning, including medical tests, but not yet in real-world scenarios. This paper investigates the use of InstructGPT to assist physicians in determining eligibility for clinical trials based on a patient's summarised medical profile. Using a prompting strategy combining one-shot, selection-inference and chain-of-thought techniques, we investigate the performance of LLMs on 10 synthetically created patient profiles. Performance is evaluated at four levels: ability to identify screenable eligibility criteria from a trial given a medical profile; ability to classify for each individual criterion whether the patient qualifies; the overall classification whether a patient is eligible for a clinical trial and the percentage of criteria to be screened by physician. We evaluated against 146 clinical trials and a total of 4,135 eligibility criteria. The LLM was able to correctly identify the screenability of 72% (2,994/4,135) of the criteria. Additionally, 72% (341/471) of the screenable criteria were evaluated correctly. The resulting trial level classification as eligible or ineligible resulted in a recall of 0.5. By leveraging LLMs with a physician-in-the-loop, a recall of 1.0 and precision of 0.71 on clinical trial level can be achieved while reducing the amount of criteria to be checked by an estimated 90%. LLMs can be used to assist physicians with pre-screening of patients for clinical trials. By forcing instruction-tuned LLMs to produce chain-of-thought responses, the reasoning can be made transparent to and the decision process becomes amenable by physicians, thereby making such a system feasible for use in real-world scenarios.

研究动机与目标

• 推动减少医生在临床试验资格筛查上的时间投入。
• 研究在给定患者资料的情况下，LLMs 是否能够从试验文本中识别可筛选的资格标准。
• 评估基于 LLM 的筛查在逐条标准和逐项试验上的分类性能。
• 评估医生在环对召回、精确度和工作量减少的影响。

提出的方法

• 使用 InstructGPT，提示结合 one-shot、选择推理（selection-inference）和 chain-of-thought 技术。
• 在 10 个合成创建的患者资料上进行测试。
• 在 146 项临床试验和 4,135 条资格标准上进行评估。
• 衡量标准可筛选性识别、逐条资格、整体试验资格，以及需要医生审查的标准所占百分比。

实验结果

研究问题

• RQ1 LLM 是否能够识别给定患者资料下哪些资格标准是可筛选的？
• RQ2 LLM 是否能够对每个可筛选标准判断患者是否符合？
• RQ3 在给定患者资料的情况下，该试验的整体资格分类（符合 vs 不符合）是多少？
• RQ4 在保持有用的召回率/精确度的前提下，可以把多少试验标准从医生转移给 LLM（即需要检查的标准数量的减少）？
• RQ5 强制链式思维推理是否提高了透明度和对医生的真实世界可用性？

主要发现

• LLM 正确识别了 72% 的标准的可筛选性（2,994/4,135）。
• LLM 正确评估了可筛选标准中 72%（341/471）。
• 试验层面的资格召回为 0.5。
• 在医生在环的情况下，试验层面的召回达到 1.0，精确度为 0.71，同时将需要检查的标准数量减少约 90%。
• 强制链式思维回答使推理变得透明，有助于现实世界的可行性。