[论文解读] PRISMA-DFLLM: An Extension of PRISMA for Systematic Literature Reviews using Domain-specific Finetuned Large Language Models

Teo Sušnjak|arXiv (Cornell University)|Jun 15, 2023

Topic Modeling被引用 9

一句话总结

PRISMA-DFLLM 提议通过领域特定微调的大模型扩展 PRISMA，以实现 AI 辅助、可扩展、持续更新的系统综述及可重复使用的微调模型。

ABSTRACT

With the proliferation of open-sourced Large Language Models (LLMs) and efficient finetuning techniques, we are on the cusp of the emergence of numerous domain-specific LLMs that have been finetuned for expertise across specialized fields and applications for which the current general-purpose LLMs are unsuitable. In academia, this technology has the potential to revolutionize the way we conduct systematic literature reviews (SLRs), access knowledge and generate new insights. This paper proposes an AI-enabled methodological framework that combines the power of LLMs with the rigorous reporting guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). By finetuning LLMs on domain-specific academic papers that have been selected as a result of a rigorous SLR process, the proposed PRISMA-DFLLM (for Domain-specific Finetuned LLMs) reporting guidelines offer the potential to achieve greater efficiency, reusability and scalability, while also opening the potential for conducting incremental living systematic reviews with the aid of LLMs. Additionally, the proposed approach for leveraging LLMs for SLRs enables the dissemination of finetuned models, empowering researchers to accelerate advancements and democratize cutting-edge research. This paper presents the case for the feasibility of finetuned LLMs to support rigorous SLRs and the technical requirements for realizing this. This work then proposes the extended PRISMA-DFLLM checklist of reporting guidelines as well as the advantages, challenges, and potential implications of implementing PRISMA-DFLLM. Finally, a future research roadmap to develop this line of AI-enabled SLRs is presented, paving the way for a new era of evidence synthesis and knowledge discovery.

研究动机与目标

• 在文献迅速增长的背景下，激发高效、可扩展的系统综述的需求。
• 提出一个 AI 驱动的框架（PRISMA-DFLLM），将领域特定微调的大模型与 PRISMA 指南相结合。
• 概述报告指南，其中包括微调数据集、LLM 训练细节以及基于 LLM 的 SLR 评估。
• 讨论领域特定大模型在证据综合中的潜在收益、挑战，以及研究路线图。

提出的方法

• 描述将领域特定微调的大模型与 PRISMA 的整合，以支持 SLR 和持续更新的综述。
• 提出微调数据集、预处理及元数据与论文文本整合的报告要素。
• 在 PRISMA-DFLLM 框架内，解释基模型选择、微调策略及评估考量。
• 为开发和评估领域特定的大模型用于学术研究提供路线图。”],
• research_questions_new_text_placeholder_needed_to_fix_quote_null_fields_initialization_not_provided?": [],
• research_questions

实验结果

研究问题

• RQ1 如何将 PRISMA 指南扩展以容纳用于 SLR 的领域特定微调大模型？
• RQ2 在 PRISMA-DFLLM 中需要哪些报告要素来记录微调数据集、LLM 训练和评估？
• RQ3 部署领域特定大模型用于持续更新的系统综述的潜在收益、局限性与影响是什么？
• RQ4 为开发和评估用于证据综合的领域特定微调大模型，需要哪些未来研究路线图？

主要发现

• 领域特定微调的大模型可实现更高效、可扩展的 SLR 工作流程。
• 如 LoRA 与 QLoRA 等 PEFT 策略可在降低微调成本的同时保留领域任务性能。
• 一个结构化的 PRISMA-DFLLM 扩展可引导关于微调数据集、模型选择和评估的报告。
• 持续更新的系统综述可受益于由微调大模型支持的增量更新。
• 该方法为向公众普及微调模型并通过 AI 驱动的 SLR 加速研究提供机会。