[Paper Review] Fine-Tuning A Large Language Model for Systematic Review Screening

Kweku Yamoah, Noah L. Schroeder+3 more|arXiv (Cornell University)|Mar 25, 2026

Meta-analysis and systematic reviews0 citations

TL;DR

The authors fine-tune a small 1.2B-parameter LLM for title/abstract screening in a single systematic review, achieving strong improvements over the baseline with high agreement to human decisions on the full dataset and consistent results across inference runs.

ABSTRACT

Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.

Motivation & Objective

• Motivate reducing screening workload in systematic reviews through model fine-tuning on human-annotated data.
• Assess whether a small LLM can achieve strong, consistent screening performance for a specific review.
• Demonstrate evaluation strategies including held-out tests and full dataset testing.
• Provide a reproducible pipeline with model, dataset, and code openly available.

Proposed method

• Fine-tune Liquid AI’s LFM2.5-1.2B-Instruct (bf16) using full-parameter supervised fine-tuning on instruction–response pairs.
• Use Unsloth with Hugging Face TRL training stack for efficient fine-tuning on a single GPU.
• Train with 320 steps, learning rate 2e-5, batch size constraints, and short warmup; max sequence length 4096.
• Format data as instruction–response pairs with response masking (loss computed on assistant tokens only).
• Partition a curated 371-abstract dataset into training (315) and test (56) with an enrichment of include cases to address class imbalance.
• Evaluate with multi-metric, imbalance-aware measures (balanced accuracy, weighted and per-class F1/F2, confusion matrix) and inter-rater reliability (Cohen’s kappa, PABAK, Gwet’s AC1, Fleiss’ kappa).
• Conduct multi-pass inference at temperatures 0.1, 0.4, 0.8 to assess consistency; parse predictions from chat-based outputs.

Experimental results

Research questions

• RQ1 What is the model’s baseline performance at title/abstract screening versus human ratings?
• RQ2 How does a fine-tuned small LLM perform on a validation dataset compared to human ratings?
• RQ3 How does the fine-tuned model perform on the full dataset compared to a human rater?
• RQ4 Is the model's screening performance robust across multiple inference temperatures?

Key findings

• Baseline (base model, full dataset) balanced accuracy 53.07% and overall accuracy 6.52%.
• Fine-tuned model on held-out test split achieves 94.64% accuracy and 94.49% balanced accuracy.
• Fine-tuned model on full dataset yields 86.40% accuracy and 88.78% balanced accuracy.
• On the full dataset, include class recall 91.18% but precision 2.69% (F1 5.22%), exclude class precision 86.38% and recall 92.67% (F1 92.67%).
• Model–human agreement on the full dataset is 86.40% with Gwet AC1 = 0.843; Cohen’s kappa is 0.045 and PABAK = 0.728.
• Three inference passes across temperatures 0.1, 0.4, 0.8 show perfect LLM-only consistency (Cohen’s kappa = 1.0 between passes) and multi-rater AC1 = 0.842 (95% CI).