[Paper Review] CTBench: A Comprehensive Benchmark for Evaluating Language Model Capabilities in Clinical Trial Design

Nafis Neehal, Bowen Wang+5 more|arXiv (Cornell University)|Jun 25, 2024

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

CTBench introduces a novel benchmark to evaluate language models (LMs) in predicting baseline features for clinical trials using metadata, leveraging two datasets—CT-Repo (1,690 trials from clinicaltrials.gov) and CT-Pub (100 trials with gold-standard features from publications). The study evaluates LM-generated baseline features using two automated methods, ListMatch-LM (GPT-4o) and ListMatch-BERT, with human-in-the-loop validation showing GPT-4o outperforms BERT at a 0.7 threshold due to superior contextual understanding.

ABSTRACT

CTBench is introduced as a benchmark to assess language models (LMs) in aiding clinical study design. Given study-specific metadata, CTBench evaluates AI models' ability to determine the baseline features of a clinical trial (CT), which include demographic and relevant features collected at the trial's start from all participants. These baseline features, typically presented in CT publications (often as Table 1), are crucial for characterizing study cohorts and validating results. Baseline features, including confounders and covariates, are also necessary for accurate treatment effect estimation in studies involving observational data. CTBench consists of two datasets: "CT-Repo," containing baseline features from 1,690 clinical trials sourced from clinicaltrials.gov, and "CT-Pub," a subset of 100 trials with more comprehensive baseline features gathered from relevant publications. Two LM-based evaluation methods are developed to compare the actual baseline feature lists against LM-generated responses. "ListMatch-LM" and "ListMatch-BERT" use GPT-4o and BERT scores (at various thresholds), respectively, for evaluation. To establish baseline results, advanced prompt engineering techniques using LLaMa3-70B-Instruct and GPT-4o in zero-shot and three-shot learning settings are applied to generate potential baseline features. The performance of GPT-4o as an evaluator is validated through human-in-the-loop evaluations on the CT-Pub dataset, where clinical experts confirm matches between actual and LM-generated features. The results highlight a promising direction with significant potential for improvement, positioning CTBench as a useful tool for advancing research on AI in CT design and potentially enhancing the efficacy and robustness of CTs.

Motivation & Objective

• To address the lack of standardized, AI-evaluated benchmarks for predicting baseline features in clinical trial design.
• To reduce bias and improve study robustness by supporting researchers in identifying relevant demographic, clinical, and confounding variables.
• To create a reproducible, scalable benchmark using real-world clinical trial data and expert-annotated gold standards.
• To evaluate the performance of state-of-the-art LMs in generating accurate baseline feature lists from minimal metadata.
• To validate evaluation methods through human-in-the-loop assessments with clinical experts.

Proposed method

• Construct two datasets: CT-Repo (1,690 RCTs from clinicaltrials.gov) and CT-Pub (100 trials with baseline features extracted from published papers via human curation).
• Develop two automated evaluation methods: ListMatch-LM (using GPT-4o to score similarity between predicted and actual features) and ListMatch-BERT (using BERT embeddings with threshold-based matching).
• Apply zero-shot and few-shot prompting with LLaMa3-70B-Instruct and GPT-4o to generate baseline feature lists from CT metadata.
• Use a fixed seed and temperature=0.0 to ensure deterministic, reproducible generation and evaluation across models.
• Validate GPT-4o’s evaluation performance via human-in-the-loop comparisons on the CT-Pub subset, where clinical experts confirm matches between LM-generated and actual features.
• Select a 0.7 threshold for BERT scoring to balance precision and recall, based on empirical evaluation across multiple thresholds.

Figure 1: Workflow of CTBench.

Experimental results

Research questions

• RQ1 Can language models accurately predict baseline features for clinical trials using only metadata, without access to full study protocols?
• RQ2 How do LLM-based evaluation methods like ListMatch-LM and ListMatch-BERT compare in measuring the similarity between predicted and actual baseline features?
• RQ3 To what extent does GPT-4o outperform BERT-based scoring in capturing nuanced semantic similarities in clinical feature descriptions?
• RQ4 How reliable are automated evaluation metrics when validated by clinical experts in a human-in-the-loop setting?
• RQ5 What are the limitations and risks of transferring societal biases from LMs into clinical trial design through baseline feature prediction?

Key findings

• GPT-4o-based evaluation (ListMatch-LM) consistently outperformed BERT-based evaluation (ListMatch-BERT) at a 0.7 threshold, indicating superior contextual understanding and similarity detection.
• The 0.7 threshold for BERT scores was selected as the optimal balance between precision and recall, minimizing false positives and false negatives in feature matching.
• Human-in-the-loop evaluations confirmed high alignment between expert-confirmed matches and GPT-4o’s predictions, validating its reliability as an evaluator.
• LLaMa3-70B-Instruct and GPT-4o both generated plausible baseline feature lists using few-shot prompting, demonstrating feasibility of AI-assisted trial design.
• The benchmark revealed that current LMs can generate relevant baseline features but require refinement to reduce omissions and overfitting to common patterns.
• Societal biases in LMs may be transferred into baseline feature predictions, potentially skewing cohort representation and undermining trial generalizability.

Figure 2: Prompt template for generation (in zero-shot setting) and evaluation