[论文解读] Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

Somin Wadhwa, Jay DeYoung+3 more|arXiv (Cornell University)|May 5, 2023

Topic Modeling参考文献 38被引用 9

一句话总结

本文将指令微调的大语言模型用于端到端条件生成，联合提取RCT摘要中的ICO（三个要素：干预、对照、结果）及推断发现，实现对比前沿方法的state-of-the-art提升，并发布可检索的结构化证据数据库。

ABSTRACT

Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable ($\sim$20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: http://ico-relations.ebm-nlp.com

研究动机与目标

• 需要从非结构化的RCT报告中提取结构化证据，以支持循证医学与AI辅助临床决策的动力与意义。
• 提出一个端到端、指令微调的LLM方法，联合从RCT摘要中提取ICO三元组、支持证据和推断标签。
• 与先前的SOTA方法比较绩效，并分析驱动模型成功与错误模式的因素。

提出的方法

• 将证据提取框定为对线性化的ICO元组（Intervention, Comparator, Outcome, Evidence, Inference label）的条件文本到文本生成任务。
• 使用教师强制的方式微调一个指令微调的LLM（Flan-T5），从摘要中产出端到端的ICO三元组及其推断。
• 在Evidence Inference数据集的仅摘要子集上训练，每个实例目标五项：ICO三元组、一个方向标签（显著增加/降低、无效）、以及一个证据片段。
• 由于输出为摘要式，采用自动评估指标与专家人工评估相结合的方式进行评估，包括与医生共同完成的人工注释研究。
• 提供对提取出来的结构化证据的原型检索界面，并发布模型权重和数据集衍生发现。

Figure 1: We fine-tune a Large Language Model (LLM) to map from free-text descriptions of clinical trials to structured representations of findings.

实验结果

研究问题

• RQ1 指令微调的LLMs 能否联合提取ICO三元组并推断RCT摘要中报道的结果方向？
• RQ2 端到端生成是否优于管线式或多阶段方法在医学证据推断中的表现？
• RQ3 在提取目标中包含证据片段对模型推断准确性有何影响？
• RQ4 模型对大量RCTs语料库的泛化能力如何，能否处理非穷尽引用？

主要发现

• 端到端生成结合指令微调的Flan-T5在对比前沿方法上获得显著提升，端到端ICO提取F1大约提高了约20个百分点。
• ICO-三元组层面的评估显示，随着模型规模增大、指令微调效果增强，Precison和Recall显著提升（例如Flan-T5-large 获得较高F1）。
• 人工专家评估显示生成的元组在相关性和正确性上达到高一致性（Fleiss kappa = 0.77），专家对相关性的达成率约为92.4%。
• 将证据片段作为目标显著支持推断任务并提升联合提取性能。
• 原型检索界面通过对来自大型RCT语料库（Trialstreamer子集）的结构化证据进行SQL检索，展示了实用性。

Figure 2: We propose instructional fine-tuning a large language model (top) using standard supervision to elicit evidence within generated ICO tuples. This approach yields substantial improvements over existing joint extraction approaches (bottom) where the entire task is decomposed into different i