[Paper Review] Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation

Ambrose Robinson, William Thorne+5 more|arXiv (Cornell University)|Aug 12, 2023

Meta-analysis and systematic reviews7 citations

TL;DR

Bio-SIEVE introduces instruction-tuned LLMs for automated literature screening in biomedical systematic reviews, outperforming both ChatGPT and traditional models by accurately classifying study relevance using detailed selection criteria. It achieves superior generalization across medical domains and enables exclusion reasoning, offering a flexible, zero-shot solution for SR automation without retraining.

ABSTRACT

Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility.

Motivation & Objective

• To address the high cost and time burden of manual literature screening in biomedical systematic reviews.
• To develop a zero-shot, instruction-tuned LLM that generalizes across diverse medical domains without retraining.
• To enable accurate inclusion/exclusion classification using complex, multi-criteria selection rules.
• To explore exclusion reasoning as a novel task to improve transparency and reduce reviewer workload.
• To evaluate multi-task learning with PICO extraction and exclusion justification for transfer benefits.

Proposed method

• Fine-tuned LLaMA and Guanaco models via instruction tuning using a curated dataset derived from Cochrane reviews.
• Constructed a dataset of abstracts labeled for inclusion/exclusion based on specific review criteria, including exclusion reasoning.
• Applied instruction tuning with detailed, natural language prompts to guide model behavior for screening and explanation tasks.
• Explored multi-task training with PICO extraction and exclusion reasoning to leverage cross-task knowledge transfer.
• Used LoRA (Low-Rank Adaptation) for efficient fine-tuning of large models on domain-specific screening tasks.
• Evaluated performance using standard NLP metrics, including F1, precision, recall, and human preference rankings.

Figure 1: A simple representation of the systematic review process depicting the stage which Bio-SIEVE aims to assist. The black funnels are the monotonous and highly resource intensive bottlenecks of the process.

Experimental results

Research questions

• RQ1 Can instruction-tuned LLMs outperform zero-shot LLMs like ChatGPT in systematic review abstract screening?
• RQ2 Does instruction tuning on domain-specific criteria improve generalization across diverse biomedical topics?
• RQ3 Can exclusion reasoning be effectively modeled as a generative task to support transparency in screening decisions?
• RQ4 Does multi-task learning with PICO extraction and exclusion reasoning improve overall screening performance?
• RQ5 How does Bio-SIEVE compare to traditional ML models in terms of sensitivity and specificity?

Key findings

• Bio-SIEVE outperformed both ChatGPT and traditional models like SVM in inclusion/exclusion classification, achieving higher F1 and recall.
• The model generalized effectively across different medical domains without fine-tuning, demonstrating strong zero-shot generalization.
• Exclusion reasoning was successfully modeled, enabling the generation of valid, contextually relevant reasons for rejecting studies.
• Multi-task training (Bio-SIEVE-Multi) showed limited transfer benefits and underperformed single-task Bio-SIEVE on screening tasks.
• Bio-SIEVE-Multi showed promise in inclusion reasoning but failed to match ChatGPT’s performance in human preference rankings.
• The model achieved high accuracy in filtering subtle irrelevancies, such as excluding muscle trauma studies from oral health reviews.

Figure 2: The topic distribution of the inclusion/exclusion classification samples in the train and test splits of the Instruct Cochrane dataset.