[论文解读] The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

Dmitry Scherbakov, Nina Hubig+3 more|arXiv (Cornell University)|Sep 6, 2024

Topic Modeling被引用 5

一句话总结

本论文综述了大型语言模型如何实现文献综述自动化，识别主导的基于GPT的架构、最自动化的评审阶段，以及数据提取与筛选的表现。

ABSTRACT

Objective: This study aims to summarize the usage of Large Language Models (LLMs) in the process of creating a scientific review. We look at the range of stages in a review that can be automated and assess the current state-of-the-art research projects in the field. Materials and Methods: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar databases by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on which uses OpenAI gpt-4o model. ChatGPT was used to clean extracted data and generate code for figures in this manuscript, ChatGPT and Scite.ai were used in drafting all components of the manuscript, except the methods and discussion sections. Results: 3,788 articles were retrieved, and 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n=126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n=26, 15.1%) were actual reviews that used LLM during their creation. Most citations focused on automation of a particular stage of review, such as Searching for publications (n=60, 34.9%), and Data extraction (n=54, 31.4%). When comparing pooled performance of GPT-based and BERT-based models, the former were better in data extraction with mean precision 83.0% (SD=10.4), and recall 86.0% (SD=9.8), while being slightly less accurate in title and abstract screening stage (Maccuracy=77.3%, SD=13.0). Discussion/Conclusion: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. The results looked promising, and we anticipate that LLMs will change in the near future the way the scientific reviews are conducted.

研究动机与目标

• 总结LLMs在创建科学综述过程中的使用方式。
• 评估文献综述的哪些阶段可以由LLMs实现自动化。
• 评述当前用于综述自动化的LLMs的最先进项目。
• 讨论未来科学综述撰写方式可能发生的变化及影响。

提出的方法

• 在2024年6月对PubMed、Scopus、Dimensions和Google Scholar进行了检索。
• 在Covidence中进行筛选与提取，配备使用OpenAI GPT-4o的LLM插件。
• 使用ChatGPT清洗提取的数据并为手稿中的图形生成代码。
• 在撰写所有手稿组成部分时使用ChatGPT及本URL，除了方法与讨论部分。
• 比较GPT-based与BERT-based模型在提取与筛选任务上的性能。

实验结果

研究问题

• RQ1 文献综述的哪些阶段可以由LLMs自动化？
• RQ2 基于LLM的综述自动化的当前最先进水平是怎样的？
• RQ3 在关键的审稿任务中，GPT基模型与BERT基模型相比如何？
• RQ4 LLMs在生成实际文献综述与自动化单个阶段方面的使用程度如何？

主要发现

• 检索到3,788篇文章。
• 有172项研究被认定有资格进入最终综述。
• ChatGPT和基于GPT的LLMs是综述自动化中最具主导性的架构（n=126，73.2%）。
• 存在大量自动化项目，但只有26篇论文（15.1%）是在创作过程中使用LLM的实际综述。
• 大多数引用集中在检索出版物（n=60，34.9%）和数据提取（n=54，31.4%）。
• GPT-based模型在数据提取方面优于BERT-based模型（平均精度83.0%，SD=10.4；召回率86.0%，SD=9.8），但在标题/摘要筛选方面略低准确度（Maccuracy=77.3%，SD=13.0）。