[Paper Review] PaperQA: Retrieval-Augmented Generative Agent for Scientific Research

Jakub Lála, Odhran O’Donoghue+4 more|arXiv (Cornell University)|Dec 8, 2023

Topic ModelingComputer Science51 citations

TL;DR

PaperQA is an agent-based Retrieval-Augmented Generation system that searches full-text papers, gathers evidence, and answers scientific questions with citations, outperforming several LLMs and commercial tools on LitQA while matching human expert performance at lower cost.

ABSTRACT

Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.

Motivation & Objective

• Address the challenge of hallucinations and irreproducibility in LLM-based scientific QA by leveraging retrieval-augmented generation (RAG).
• Develop an agent-based RAG workflow that can iteratively search, gather evidence from full-text papers, and generate grounded answers with provenance.
• Introduce LitQA, a full-text scientific QA benchmark, to evaluate retrieval and synthesis from contemporary literature.
• Demonstrate that PaperQA outperforms existing LLMs and commercial tools on LitQA and approaches human expert performance in both accuracy and efficiency.

Proposed method

• Decompose RAG into modular tools (search, gather_evidence, answer_question) executed by an agent LLM.
• Use map-reduce style evidence gathering to consider information from multiple sources and provide intermediate evidence before final answer.
• Assign numerical relevance scores to text chunks via LLM-generated scores in addition to vector similarities.
• Integrate a priori and a posteriori prompting to exploit latent knowledge and improve retrieval decisions.
• Retrieve 4,000-character chunks from full-text papers and embed with text-embedding-ada-002, stored in a vector database for retrieval.
• Enable iterative tool usage (search and gather_evidence) until sufficient evidence accumulates before answering.
• Evaluate using LitQA and standard QA benchmarks; compare against GPT-4, Claude-2, AutoGPT, and commercial tools; assess citation hallucination rates.

Experimental results

Research questions

• RQ1 Can a modular, agent-based RAG architecture improve factual grounding and reduce hallucinations in scientific QA compared to fixed, linear RAG pipelines?
• RQ2 How well does PaperQA retrieve and synthesize information from full-text scientific papers to answer questions that require cross-paper reasoning?
• RQ3 What is the performance and cost trade-off of PaperQA relative to human experts and commercial science QA tools on LitQA and standard benchmarks?
• RQ4 Does PaperQA exhibit fewer citation hallucinations and better uncertainty handling than contemporary LLM-based QA systems?

Key findings

• PaperQA outperforms GPT-4 and other commercial tools on LitQA and matches human performance on the benchmark in both accuracy and response time, while being cheaper.
• PaperQA demonstrates a low rate of citation hallucinations (0% in tested cases) compared with other LLMs that exhibit substantial hallucination rates.
• On standard QA benchmarks, PaperQA with PaperQA-specific prompts and retrieval shows competitive or superior performance relative to strong baselines, including GPT-4 alone.
• LitQA requires retrieval and synthesis from full-text papers; PaperQA shows strong retrieval performance and effective multi-source evidence aggregation via its map-reduce style workflow.
• Ablation studies show the importance of multiple components (multi-round search, evidence gathering, and leveraging the LLM’s latent knowledge through an ask step) for best performance.
• PaperQA is cost-efficient, averaging about $0.18 per question for GPT-4/GPT-3.5-turbo configurations and approximately 2.4 hours per batch of questions (comparable to human time).