[论文解读] LitLLM: A Toolkit for Scientific Literature Review

Shubham Agarwal, Sahu, Gaurav+6 more|arXiv (Cornell University)|Feb 2, 2024

Biomedical Text Mining and Ontologies被引用 9

一句话总结

LitLLM 提供一个模块化的检索增强生成（RAG）工具包，从用户提供的摘要开始生成文献综述，使用关键字驱动的检索、基于LLM的重新排序，以及基于计划的生成，以减少幻觉。

ABSTRACT

Conducting literature reviews for scientific papers is essential for understanding research, its limitations, and building on existing work. It is a tedious task which makes an automatic literature review generator appealing. Unfortunately, many existing works that generate such reviews using Large Language Models (LLMs) have significant limitations. They tend to hallucinate-generate non-factual information-and ignore the latest research they have not been trained on. To address these limitations, we propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles, specialized prompting and instructing techniques with the help of LLMs. Our system first initiates a web search to retrieve relevant papers by summarizing user-provided abstracts into keywords using an off-the-shelf LLM. Authors can enhance the search by supplementing it with relevant papers or keywords, contributing to a tailored retrieval process. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Finally, the related work section is generated based on the re-ranked results and the abstract. There is a substantial reduction in time and effort for literature review compared to traditional methods, establishing our toolkit as an efficient alternative. Our project page including the demo and toolkit can be accessed here: https://litllm.github.io

研究动机与目标

• 从摘要或研究想法出发，实现文献综述生成的自动化和加速。
• 将生成内容基于检索到的、最新的论文，从而减少幻觉。
• 通过句子计划和模块化组件提供可控的生成过程。

提出的方法

• 将输入摘要摘要成用于通过 Semantic Scholar API 进行学术检索的关键词查询。
• 应用基于LLM的再排序器按与查询摘要的相关性对检索到的论文进行排序。
• 使用在重新排序后的论文和摘要条件下的LLM生成相关工作部分。
• 结合基于句子计划的提示，控制生成文本的长度和结构。
• 采用 Retrieval Augmented Generation (RAG) 将输出基于检索到的文献。

Figure 1: LitLLM on Huggingface Space. Our system works on the Retrieval Augmented Generation (RAG) principle to generate the literature review grounded in retrieved relevant papers. User needs to provide the abstract in the textbox (in purple) and press send to get the generated related work (in re

实验结果

研究问题

• RQ1 基于 RAG 的模块化流水线是否能够从摘要生成在事实层面有据可依的相关工作部分？
• RQ2 与零样本生成相比，重新排序是否提高了生成的文献综述的事实依据和相关性？
• RQ3 基于句子计划的生成是否能够提供可控且简洁、符合用户需求的文献综述？

主要发现

• 一个模块化的 LitLLM 流水线使用基于关键词的搜索查询、一个重新排序器以及一个 LLM 生成器来产生基于检索论文的相关工作。
• RAG 通过将生成与检索源绑定并提供最新内容来减少幻觉。
• 句子计划提示提供可控、简洁的生成，而零样本生成则提供更广泛、信息性覆盖。
• 该系统支持通过可选的种子论文或关键词进行交互式搜索，以定制检索集合。
• 该工具强调归属并提醒可能的幻觉风险，并计划在未来工作中获取整篇论文。

Figure 2: Schematic diagram of the modular pipeline used in our system. In the default setup, we summarize the research abstract into a keyword query, which is used to retrieve relevant papers from an academic search engine. We use an LLM-based reranker to select the most relevant paper relative to