[Paper Review] Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

Somin Wadhwa, Jay DeYoung+3 more|arXiv (Cornell University)|May 5, 2023

Topic Modeling38 references9 citations

TL;DR

This paper trains instruction-tuned LLMs to jointly extract ICO (Interventions, Comparators, Outcomes) and inferred findings from RCT abstracts as end-to-end conditional generation, achieving state-of-the-art gains over prior methods and releasing a searchable structured-evidence database.

ABSTRACT

Results from Randomized Controlled Trials (RCTs) establish the comparative effectiveness of interventions, and are in turn critical inputs for evidence-based care. However, results from RCTs are presented in (often unstructured) natural language articles describing the design, execution, and outcomes of trials; clinicians must manually extract findings pertaining to interventions and outcomes of interest from such articles. This onerous manual process has motivated work on (semi-)automating extraction of structured evidence from trial reports. In this work we propose and evaluate a text-to-text model built on instruction-tuned Large Language Models (LLMs) to jointly extract Interventions, Outcomes, and Comparators (ICO elements) from clinical abstracts, and infer the associated results reported. Manual (expert) and automated evaluations indicate that framing evidence extraction as a conditional generation task and fine-tuning LLMs for this purpose realizes considerable ($\sim$20 point absolute F1 score) gains over the previous SOTA. We perform ablations and error analyses to assess aspects that contribute to model performance, and to highlight potential directions for further improvements. We apply our model to a collection of published RCTs through mid-2022, and release a searchable database of structured findings: http://ico-relations.ebm-nlp.com

Motivation & Objective

• Motivate the need for structured evidence extraction from unstructured RCT reports to support evidence-based medicine and AI-assisted clinical decisions.
• Propose an end-to-end, instruction-tuned LLM approach to jointly extract ICO triplets, supporting evidence, and inference labels from RCT abstracts.
• Evaluate performance against prior SOTA methods and analyze what drives model success and error patterns.

Proposed method

• Frame evidence extraction as a conditional text-to-text generation task over linearized ICO tuples (Intervention, Comparator, Outcome, Evidence, Inference label).
• Fine-tune an instruction-tuned LLM (Flan-T5) with teacher forcing to generate end-to-end ICO triplets and their inferences from abstracts.
• Train on an abstract-only subset of the Evidence Inference dataset with five targets per instance: ICO triplet, a direction label (significant increase/decrease, no effect), and an evidence snippet.
• Evaluate with both automated metrics and expert human judgments due to abstractive outputs, including a manual annotation study with medical doctors.
• Provide a prototype search interface over the extracted structured evidence and release the model weights and dataset-derived findings.

Figure 1: We fine-tune a Large Language Model (LLM) to map from free-text descriptions of clinical trials to structured representations of findings.

Experimental results

Research questions

• RQ1 Can instruction-tuned LLMs jointly extract ICO triplets and infer the reported outcome direction from RCT abstracts?
• RQ2 Does end-to-end generation outperform pipeline or multi-stage approaches for medical evidence inference?
• RQ3 What is the impact of including evidence spans in the extraction targets on and the model's inference accuracy?
• RQ4 How well does the model generalize to a large corpus of RCTs and handle non-exhaustive references?

Key findings

• End-to-end generation with instruction-tuned Flan-T5 achieves substantial gains over prior SOTA, approximately a 20 point absolute improvement in F1 for end-to-end ICO extraction.
• ICO-triplet level evaluation shows strong precision and recall improvements with larger, instruction-tuned models (e.g., Flan-T5-large yielding high F1).
• Human expert evaluation reports high agreement (Fleiss kappa = 0.77) on relevance and correctness of generated tuples, with experts agreeing on relevance ~92.4% of the time.
• Including evidence spans as targets significantly supports the inference task and improves joint extraction performance.
• A prototype search interface demonstrates practical utility by enabling SQL-based searches over the structured evidence extracted from a large RCT corpus (Trialstreamer subset).

Figure 2: We propose instructional fine-tuning a large language model (top) using standard supervision to elicit evidence within generated ICO tuples. This approach yields substantial improvements over existing joint extraction approaches (bottom) where the entire task is decomposed into different i