[Paper Review] Zero-shot Generative Large Language Models for Systematic Review Screening Automation

Shuai Wang, Harrisen Scells+4 more|arXiv (Cornell University)|Jan 12, 2024

Topic Modeling4 citations

TL;DR

This paper proposes a zero-shot generative LLM approach for automating systematic review document screening, using instruction-tuned models and a novel calibration technique to meet target recall levels. It demonstrates that calibrated ensembles of zero-shot LLMs outperform state-of-the-art fine-tuned models like Bio-SIEVE while eliminating the need for costly labeling or fine-tuning, significantly reducing manual screening effort.

ABSTRACT

Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches.

Motivation & Objective

• To automate systematic review document screening using zero-shot generative LLMs without task-specific fine-tuning or labeling.
• To evaluate the impact of LLM architecture, instruction fine-tuning, and calibration on screening effectiveness.
• To assess whether ensemble methods combining multiple LLMs and a BERT baseline can outperform existing state-of-the-art approaches.
• To determine if zero-shot LLMs can reliably achieve target recall levels required in clinical systematic reviews.

Proposed method

• Employ zero-shot generative LLMs to classify documents as 'included' or 'excluded' based on prompt instructions derived from review topics.
• Use the probability of the next predicted tokens ('yes' or 'no') to make classification decisions, with a calibrated threshold θ to control recall.
• Calibrate the decision boundary θ using a predefined recall target, derived from seed documents or prior reviews, to balance precision and recall.
• Form an ensemble by combining predictions from multiple top-performing LLMs (e.g., LlaMa2-7b-ins, LlaMa2-13b-ins) and a BioBERT baseline to improve robustness and performance.
• Apply the calibration technique to both zero-shot models and the fine-tuned Bio-SIEVE baseline for comparative evaluation.
• Evaluate all methods on five standard test collections (e.g., CLEF-TAR) using standard metrics: B-AC, WSS, precision, F3, and recall.

Experimental results

Research questions

• RQ1 RQ1: How does the architecture and size of LLMs influence screening effectiveness?
• RQ2 RQ2: How does instruction-based fine-tuning affect performance compared to base models?
• RQ3 RQ3: How does output calibration with a threshold θ improve recall and overall effectiveness?
• RQ4 RQ4: How does ensembling multiple LLMs and neural baselines impact performance compared to individual models?

Key findings

• LlaMa2-7b-ins outperformed larger LlaMa2-13b-ins, indicating that model size alone does not guarantee better performance for this task.
• Instruction fine-tuning consistently improved performance over base models, demonstrating its importance for effective zero-shot application.
• Calibration with a threshold θ significantly improved recall and precision, enabling reliable achievement of target recall levels.
• The calibrated ensemble of LlaMa2-7b-ins, LlaMa2-13b-ins, and BioBERT achieved higher WSS, precision, and F3 than any individual model, including the fine-tuned Bio-SIEVE.
• The ensemble method outperformed Bio-SIEVE in both B-AC and WSS under the same calibration conditions, despite Bio-SIEVE being fine-tuned on the same datasets.
• The calibrated zero-shot approach achieved high recall across diverse topics, suggesting practical viability for real-world systematic review workflows.