[Paper Review] Artificial Intelligence for Literature Reviews: Opportunities and Challenges

Francisco Bolaños, Angelo A. Salatino+2 more|arXiv (Cornell University)|Feb 13, 2024

Biomedical Text Mining and Ontologies14 citations

TL;DR

A comprehensive survey of how AI, especially large language models, semi-automates screening and extraction in systematic literature reviews, analyzing 21 SLR tools and 11 AI features.

ABSTRACT

This manuscript presents a comprehensive review of the use of Artificial Intelligence (AI) in Systematic Literature Reviews (SLRs). A SLR is a rigorous and organised methodology that assesses and integrates previous research on a given topic. Numerous tools have been developed to assist and partially automate the SLR process. The increasing role of AI in this field shows great potential in providing more effective support for researchers, moving towards the semi-automatic creation of literature reviews. Our study focuses on how AI techniques are applied in the semi-automation of SLRs, specifically in the screening and extraction phases. We examine 21 leading SLR tools using a framework that combines 23 traditional features with 11 AI features. We also analyse 11 recent tools that leverage large language models for searching the literature and assisting academic writing. Finally, the paper discusses current trends in the field, outlines key research challenges, and suggests directions for future research.

Motivation & Objective

• Assess how AI techniques are applied to semi-automate SLR screening and extraction stages.
• Introduce a framework combining traditional 23 features with 11 AI-specific features.
• Review 21 leading AI-enhanced SLR tools and 11 recent LLM-based tools.
• Identify current trends, challenges, and opportunities for AI in SLRs.

Proposed method

• Conduct a PRISMA-guided systematic review of AI-enhanced SLR tools.
• Apply inclusion/exclusion criteria to select tools that semi-automate screening or extraction.
• Develop a framework with 23 general features and 11 AI-specific features to analyze tools.
• Perform tool-level analysis using literature, official docs, and developer outreach.
• Synthesize findings on trends, challenges, and future research directions.

Experimental results

Research questions

• RQ1 How are AI techniques currently used to semi-automate the screening and extraction phases of SLRs?
• RQ2 What features (general and AI-specific) characterize AI-enhanced SLR tools?
• RQ3 What are the main trends, challenges, and future directions in AI for systematic literature reviews?

Key findings

• Nineteen tools use AI for screening; four tools support extraction; Iris.ai and RobotReviewer/RobotSearch cover both stages.
• An expanded framework with 11 AI features and 23 general features was used to evaluate tools.
• Most tools perform paper classification (relevant/irrelevant) using classifiers like SVM; some use embeddings or topic maps.
• LLM-based tools for searching and writing are emerging and may be integrated with SLR tools in the future.
• The study highlights gaps in AI feature coverage and calls for more comprehensive analyses of AI capabilities in SLR tools.