[Paper Review] LitLLM: A Toolkit for Scientific Literature Review

Shubham Agarwal, Sahu, Gaurav+6 more|arXiv (Cornell University)|Feb 2, 2024

Biomedical Text Mining and Ontologies9 citations

TL;DR

LitLLM presents a modular Retrieval Augmented Generation (RAG) toolkit that generates literature reviews starting from a user-provided abstract, using keyword-driven retrieval, LLM-based re-ranking, and plan-based generation to reduce hallucinations.

ABSTRACT

Conducting literature reviews for scientific papers is essential for understanding research, its limitations, and building on existing work. It is a tedious task which makes an automatic literature review generator appealing. Unfortunately, many existing works that generate such reviews using Large Language Models (LLMs) have significant limitations. They tend to hallucinate-generate non-factual information-and ignore the latest research they have not been trained on. To address these limitations, we propose a toolkit that operates on Retrieval Augmented Generation (RAG) principles, specialized prompting and instructing techniques with the help of LLMs. Our system first initiates a web search to retrieve relevant papers by summarizing user-provided abstracts into keywords using an off-the-shelf LLM. Authors can enhance the search by supplementing it with relevant papers or keywords, contributing to a tailored retrieval process. Second, the system re-ranks the retrieved papers based on the user-provided abstract. Finally, the related work section is generated based on the re-ranked results and the abstract. There is a substantial reduction in time and effort for literature review compared to traditional methods, establishing our toolkit as an efficient alternative. Our project page including the demo and toolkit can be accessed here: https://litllm.github.io

Motivation & Objective

• Automate and accelerate literature review generation starting from an abstract or research idea.
• Ground generated content in retrieved, up-to-date papers to reduce hallucinations.
• Provide a controllable generation process via sentence plans and modular components.

Proposed method

• Summarize the input abstract into keyword queries for academic search via Semantic Scholar APIs.
• Apply an LLM-based re-ranker to order retrieved papers by relevance to the query abstract.
• Generate the related work section using an LLM conditioned on the re-ranked papers and the abstract.
• Incorporate sentence-plan based prompts to control the length and structure of the generated text.
• Adopt Retrieval Augmented Generation (RAG) to ground the output in retrieved documents.

Figure 1: LitLLM on Huggingface Space. Our system works on the Retrieval Augmented Generation (RAG) principle to generate the literature review grounded in retrieved relevant papers. User needs to provide the abstract in the textbox (in purple) and press send to get the generated related work (in re

Experimental results

Research questions

• RQ1 Can a RAG-based, modular pipeline produce factually grounded related work sections from an abstract?
• RQ2 Does re-ranking improve factual grounding and relevance of generated literature reviews compared to zero-shot generation?
• RQ3 Does sentence-plan based generation offer controllable and succinct literature reviews tailored to a user’s needs?

Key findings

• A modular LitLLM pipeline uses keyword-derived search queries, a re-ranker, and an LLM generator to produce related work grounded in retrieved papers.
• RAG reduces hallucinations by grounding generation in retrieved sources and enabling up-to-date content.
• Sentence-plan prompting provides controllable, succinct generation, while zero-shot generation offers broader, informative coverage.
• The system supports interactive search via optional seed papers or keywords to tailor the retrieved set.
• The tool emphasizes attribution and cautions about possible hallucinations, with plans to ingest full papers in future work.

Figure 2: Schematic diagram of the modular pipeline used in our system. In the default setup, we summarize the research abstract into a keyword query, which is used to retrieve relevant papers from an academic search engine. We use an LLM-based reranker to select the most relevant paper relative to