[Paper Review] AutoTrial: Prompting Language Models for Clinical Trial Design

Zifeng Wang, Cao Xiao+1 more|arXiv (Cornell University)|May 19, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

AutoTrial is a fine-tuned language model that generates clinically accurate, fluent, and coherent eligibility criteria for clinical trials using instruction tuning, retrieval-augmented generation (RAG), and multi-step reasoning supervision. It achieves a 60% human evaluation win rate over GPT-3.5 while using significantly fewer parameters, demonstrating strong performance in controllable, knowledge-augmented, and interpretable clinical trial design.

ABSTRACT

Clinical trials are critical for drug development. Constructing the appropriate eligibility criteria (i.e., the inclusion/exclusion criteria for patient recruitment) is essential for the trial's success. Proper design of clinical trial protocols should consider similar precedent trials and their eligibility criteria to ensure sufficient patient coverage. In this paper, we present a method named AutoTrial to aid the design of clinical eligibility criteria using language models. It allows (1) controllable generation under instructions via a hybrid of discrete and neural prompting, (2) scalable knowledge incorporation via in-context learning, and (3) explicit reasoning chains to provide rationales for understanding the outputs. Experiments on over 70K clinical trials verify that AutoTrial generates high-quality criteria texts that are fluent and coherent and with high accuracy in capturing the relevant clinical concepts to the target trial. It is noteworthy that our method, with a much smaller parameter size, gains around 60% winning rate against the GPT-3.5 baselines via human evaluations.

Motivation & Objective

• Address the high failure rate of clinical trials due to suboptimal eligibility criteria design, which often leads to recruitment issues and costly amendments.
• Reduce the time and cost of trial protocol development by automating the generation of high-quality eligibility criteria using language models.
• Enable controllable, knowledge-grounded, and interpretable generation of clinical trial criteria through instruction tuning and explicit reasoning supervision.
• Minimize reliance on large-scale retraining by supporting incremental knowledge updates through neural prompting and external retrieval.
• Improve clinical relevance and accuracy of generated criteria by incorporating precedent trial data and domain-specific knowledge via retrieval-augmented generation.

Proposed method

• Fine-tune a language model using instruction tuning to generate eligibility criteria that precisely follow user-specified instructions regarding target conditions and treatments.
• Integrate retrieval-augmented generation (RAG) with an external dense retriever to dynamically incorporate relevant eligibility criteria from prior successful trials during generation.
• Employ neural prompting to maintain and update internal knowledge representations, enabling efficient incremental learning without full retraining.
• Apply multi-step reasoning supervision to generate explicit rationales for each generated criterion, enhancing interpretability and clinical trustworthiness.
• Use a hybrid prompting framework combining discrete instruction templates and continuous neural prompts to balance control and fluency in generation.
• Train the model end-to-end on a large-scale dataset of 70K+ clinical trial protocols, with data split into instruction types for incremental learning and evaluation.

Figure 1: The workflow of the proposed AutoTrial . Step I: pre-train on unlabeled trial documents with prompts to mimic the multi-step reasoning. Step II: finetune the model to generate criteria under instructions. Step III: generate diverse target criteria by instructions with large-scale sampling

Experimental results

Research questions

• RQ1 Can a fine-tuned language model generate clinically accurate, fluent, and coherent eligibility criteria for clinical trials using only instruction and retrieval-based knowledge?
• RQ2 How does the integration of retrieval-augmented generation and multi-step reasoning supervision improve the quality and interpretability of generated clinical criteria?
• RQ3 To what extent can a smaller, instruction-tuned model outperform larger, general-purpose LLMs like GPT-3.5 in clinical trial design tasks?
• RQ4 How effective is the incremental learning mechanism in maintaining performance when new instruction types are introduced without full retraining?
• RQ5 What is the relative contribution of each component—reasoning supervision, RAG, and neural prompting—to the overall performance of the model?

Key findings

• AutoTrial achieves a clinical accuracy F1 score of 0.91, precision of 0.91, recall of 0.92, and Jaccard score of 0.84, significantly outperforming all baselines, which scored below 0.5 on these metrics.
• In human evaluations, AutoTrial achieved a 60% winning rate over GPT-3.5-turbo-0301 on trial design tasks, demonstrating strong performance despite using a much smaller parameter count.
• The ablation study shows that removing RAG or neural prompting leads to a significant drop in performance, confirming their critical role in knowledge integration and generation control.
• The incremental learning variant of AutoTrial mitigates catastrophic forgetting and maintains strong performance even after updating on four times more data, though it lags behind the retrained baseline over time.
• Multi-step reasoning supervision improves model interpretability and balances performance across inclusion and exclusion criteria, with only a minor drop in exclusion criterion quality compared to the full model.
• The model demonstrates scalability and efficiency in knowledge incorporation, supporting continuous updates via neural prompting without full retraining, making it suitable for real-world deployment in evolving clinical databases.