[论文解读] CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study

Zihan Guan, Zihao Wu+6 more|arXiv (Cornell University)|Jul 21, 2023

Artificial Intelligence in Healthcare and Education被引用 15

一句话总结

CohortGPT 使用知识图谱和强化学习驱动的动态 Chain-of-Thought 采样，在 few-shot 设置中改善基于 LLM 的医学文本分类，用于临床研究参与者招募。

ABSTRACT

Participant recruitment based on unstructured medical texts such as clinical notes and radiology reports has been a challenging yet important task for the cohort establishment in clinical research. Recently, Large Language Models (LLMs) such as ChatGPT have achieved tremendous success in various downstream tasks thanks to their promising performance in language understanding, inference, and generation. It is then natural to test their feasibility in solving the cohort recruitment task, which involves the classification of a given paragraph of medical text into disease label(s). However, when applied to knowledge-intensive problem settings such as medical text classification, where the LLMs are expected to understand the decision made by human experts and accurately identify the implied disease labels, the LLMs show a mediocre performance. A possible explanation is that, by only using the medical text, the LLMs neglect to use the rich context of additional information that languages afford. To this end, we propose to use a knowledge graph as auxiliary information to guide the LLMs in making predictions. Moreover, to further boost the LLMs adapt to the problem setting, we apply a chain-of-thought (CoT) sample selection strategy enhanced by reinforcement learning, which selects a set of CoT samples given each individual medical report. Experimental results and various ablation studies show that our few-shot learning method achieves satisfactory performance compared with fine-tuning strategies and gains superb advantages when the available data is limited. The code and sample dataset of the proposed CohortGPT model is available at: https://anonymous.4open.science/r/CohortGPT-4872/

研究动机与目标

• 解决从非结构化临床笔记和放射科报告中识别合格参与者的挑战。
• 利用领域知识图谱来引导 LLM 推理以进行疾病标签预测。
• 使用强化学习优化的动态 CoT 样本选择策略，提升 few-shot 学习性能。
• 展示在有限标注数据下，与微调方法的竞争力，并提供可重复的代码/数据集。
• 展示框架向其他医学 NLP 任务的可扩展性。

提出的方法

• 通过知识图提示嵌入医学知识（KG-as-Tree, KG-as-Relation, KG-as-Rules）到 LLM 输入中。
• 采用基于策略梯度的动态 CoT 样本选择，为每个医疗报告选择 CoT 提示。
• 使用基于 BioGPT 的状态编码器来计算报告与 CoT 候选之间的相似性，以告知策略。
• 优化奖励函数，反映正确/错误标签预测以训练 CoT 选择器。
• 以 KG-to-prompt 转换的 KG-as-Rule 为默认基线，基于消融结果。
• 在 IU-RR 和 MIMIC-CXR 数据集上，与微调基线（BioBERT, BioGPT）和 few-shot LLM 基线（Alpaca, BloomZ, ChatGPT, GPT-4）进行比较评估。

Figure 1: A policy model will be trained on a small number of training samples to dynamically select CoT samples from a CoT candidate pool. A knowledge graph containing the hierarchical information of the disease labels will be transformed into a series of executable rules. Then the dynamic CoT samp

实验结果

研究问题

• RQ1 RQ1: 在 few-shot 设置下，与微调方法相比，所提方法在医学报告分类中的表现如何？
• RQ2 RQ2: 不同的 KG-to-prompt 策略和 CoT 样本选择方法如何影响性能？
• RQ3 RQ3: 通过所选的 CoT 样本，是否能像案例研究所示那样提供可解释的逐步推理？

主要发现

• 在低数据情形下，使用 ChatGPT 或 GPT-4 的方法优于传统微调（IU-RR: 0.69 F1 with 5-shot ChatGPT; 0.81 with 5-shot GPT-4 vs BioBERT 0.44 and BioGPT 0.25）。
• 在 IU-RR 和 MIMIC-CXR 上，少样本 LLM 相对于微调在数据稀缺时显示出优势， larger splits 上也呈现相似趋势。
• KG-as-Rule 产生了最佳的整体提示性能（Exact Match 0.56, Precision 0.73, Recall 0.72, F1 0.69, HL 0.04）。
• Dynamic CoT 样本选择在多项指标上持续优于 Random、Manual 和 Most-Similar CoT 策略。
• 随着训练样本增多和更大 CoT 候选池的扩大，性能提升；观察到最优的 k-shot 值在 5 或 8 附近；过多的 CoT 样本可能损害性能。
• 案例研究表明所选 CoT 样本使 ChatGPT 能通过多步推理推断出答案，符合设计。

Figure 2: A knowledge graph was created by [ 29 ] to represent relationships between diseases, organs, or tissues. In this graph, disease labels are represented by nodes in solid boxes, corresponding organs or tissues are represented by nodes in dotted boxes, and the edges linking the nodes represen