[Paper Review] How to Read Less: Better Machine Assisted Reading Methods for Systematic Literature Reviews.

Zhe Yu, Nicholas A. Kraft+1 more|arXiv (Cornell University)|Dec 10, 2016

Software Engineering Research28 references10 citations

TL;DR

This paper proposes FASTREAD, a novel active learning method that accelerates systematic literature reviews by prioritizing documents for human review based on refactored techniques from evidence-based medicine and legal e-discovery. It reduces the number of papers researchers must read by an order of magnitude—achieving 95% recall after reviewing 20–50% fewer studies than state-of-the-art methods while maintaining the same recall of relevant primary studies.

ABSTRACT

Literature reviews can be time-consuming and tedious to complete. By cataloging and refactoring three state-of-the-art active learning techniques from evidence-based medicine and legal electronic discovery, this paper finds and implements FASTREAD, a faster technique for studying a large corpus of documents. This paper assesses FASTREAD using datasets generated from existing SE literature reviews (Hall, Wahono, Radjenovic, Kitchenham et al.). Compared to manual methods, FASTREAD lets researchers find 95% relevant studies after reviewing an order of magnitude fewer papers. Compared to other state-of-the-art automatic methods, FASTREAD reviews 20-50% fewer studies while finding same number of relevant primary studies in a systematic literature review.

Motivation & Objective

• To address the time-consuming and tedious nature of systematic literature reviews in software engineering.
• To reduce the number of documents researchers must manually review while maintaining high recall of relevant studies.
• To adapt and refactor active learning techniques from evidence-based medicine and legal e-discovery for systematic literature reviews.
• To evaluate the performance of the proposed method on real-world SE literature review datasets.
• To demonstrate that FASTREAD achieves higher efficiency than both manual and state-of-the-art automatic review methods.

Proposed method

• FASTREAD integrates and refactors three state-of-the-art active learning techniques from evidence-based medicine and legal electronic discovery.
• It uses uncertainty sampling and query strategies to prioritize documents most likely to improve model performance.
• The method dynamically selects documents for human review based on model confidence and expected information gain.
• It applies a hybrid ranking strategy that combines model uncertainty with relevance prediction to reduce redundant reviews.
• The system iteratively re-trains the model after each human labeling round, refining selection accuracy over time.
• The approach is evaluated on real datasets derived from existing SE literature reviews (Hall, Wahono, Radjenovic, Kitchenham et al.).

Experimental results

Research questions

• RQ1 Can FASTREAD reduce the number of documents a researcher must manually review while maintaining high recall in systematic literature reviews?
• RQ2 How does FASTREAD compare to manual review in terms of efficiency and recall?
• RQ3 Does FASTREAD outperform existing state-of-the-art automatic methods in reducing the number of reviewed documents for the same recall?
• RQ4 How effective is the integration of active learning techniques from medicine and law in the context of software engineering literature reviews?
• RQ5 What is the impact of document prioritization strategy on the speed and accuracy of literature review completion?

Key findings

• FASTREAD reduces the number of papers researchers must review by an order of magnitude compared to manual methods, achieving 95% recall with significantly fewer documents.
• Compared to other state-of-the-art automatic methods, FASTREAD reviews 20–50% fewer studies while achieving the same number of relevant primary studies.
• The method maintains high recall even when the number of reviewed documents is drastically reduced, demonstrating strong efficiency gains.
• The integration of active learning techniques from evidence-based medicine and legal e-discovery proves effective in software engineering literature reviews.
• The approach enables researchers to reach high recall levels with substantially less manual effort, making systematic literature reviews more scalable and feasible.