Skip to main content
QUICK REVIEW

[论文解读] OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs

Akari Asai, Jacqueline He|arXiv (Cornell University)|Nov 21, 2024
Semantic Web and OntologiesComputer Science被引用 5
一句话总结

tldr: OpenScholar 是一个检索增强型语言模型,拥有一个 4500万篇开放获取论文的数据存储,及迭代自我反馈,能够带引用地综合文献;在正确性和引用准确性方面超越若干基线。它还展示了专家偏好的输出,并对其生态系统开放源代码。

ABSTRACT

Scientific progress depends on researchers' ability to synthesize the growing body of literature. Can large language models (LMs) assist scientists in this task? We introduce OpenScholar, a specialized retrieval-augmented LM that answers scientific queries by identifying relevant passages from 45 million open-access papers and synthesizing citation-backed responses. To evaluate OpenScholar, we develop ScholarQABench, the first large-scale multi-domain benchmark for literature search, comprising 2,967 expert-written queries and 208 long-form answers across computer science, physics, neuroscience, and biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o hallucinates citations 78 to 90% of the time, OpenScholar achieves citation accuracy on par with human experts. OpenScholar's datastore, retriever, and self-feedback inference loop also improves off-the-shelf LMs: for instance, OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations, experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's 32%. We open-source all of our code, models, datastore, data and a public demo.

研究动机与目标

  • 解决在大规模范围内合成科学文献、并提供可靠署名与最新引用的挑战。
  • 开发专门面向科学领域的存储库和检索系统,以支持开放检索生成。
  • 训练并评估一个开放、 高效的 LM(8B),并通过自反馈与迭代检索来展示改进。
  • 创建 ScholarQABench,一个包含专家注释的大型跨领域长篇文献综合基准测试。

提出的方法

  • 构建 OpenScholar,配备专用的 OpenScholar-DataStore(OSDS),包含 45 百万篇论文和 234 百万条段落。
  • 使用在 peS2o 数据集上训练的双编码检索器与一个跨编码再排序器,以生成用于综合的前 N 条段落集合。
  • 使用一个语言模型生成器,先给出带内联引用的初步答案,然后进行自生成的反馈和迭代检索以完善输出。
  • 加入引用验证步骤,确保陈述由检索到的段落支撑。
  • 通过同一推理流水线生成的合成数据来训练一个更小的 8B OpenScholar-LM,结合数据筛选与与通用科学数据的混合。
Figure 1: (Top) Overview of OpenScholar : OpenScholar consists of a specialized datastore, retrievers and LMs and iteratively improves responses using self-feedback inference with retrieval. (Middle) Overview of ScholarQABench : ScholarQABench consists of 2.2k expert-written questions across multipl
Figure 1: (Top) Overview of OpenScholar : OpenScholar consists of a specialized datastore, retrievers and LMs and iteratively improves responses using self-feedback inference with retrieval. (Middle) Overview of ScholarQABench : ScholarQABench consists of 2.2k expert-written questions across multipl

实验结果

研究问题

  • RQ1检索增强的语言模型是否能够在跨多个领域的长篇输出中提供带有准确、可引用证据的文献合成?
  • RQ2在 ScholarQABench 上,OpenScholar 与专有模型和开源模型(如 GPT-4o、PaperQA2)在正确性和引用准确性方面有何差异?
  • RQ3迭代自反馈与检索对答案质量和引用可靠性有何影响?
  • RQ4当作为增强组件使用时,OpenScholar 流程是否能提升现成语言模型的性能?
  • RQ5ScholarQABench 基准在跨领域多论文文献综合评估中是否有效?

主要发现

  • OpenScholar-8B 在 ScholarQABench 的正确性上比 GPT-4o 高出 5%,比 PaperQA2 高出 7%。
  • GPT-4o 在引用上会出现 78–90% 的错觉(伪引用),而 OpenScholar 的引用准确性与人类专家相当。
  • 将 OpenScholar 与 GPT-4o 结合,相比单独使用 GPT-4o,正确性提高了 12%。
  • 专家在比较中更偏好 OpenScholar 的输出:OpenScholar-8B 为 51%,OpenScholar-GPT4o 为 70%,相比之下 GPT-4o 为 32% 的比较。
  • OpenScholar 生态系统——包括数据存储、检索器、LM 检查点和 ScholarQABench——已开源并提供公开演示。
  • 在 OpenScholar 中使用更小的检索器可以降低成本并提高检索效率,同时不牺牲性能。
Figure 2: Detailed overview of OpenScholar inference (top) and training (bottom) . At inference time, given an input $x$ , OpenScholar first uses a retriever to identify relevant papers from a specialized datastore ( OpenScholar -Datastore), and then uses a reranker to refine and identify the top $N
Figure 2: Detailed overview of OpenScholar inference (top) and training (bottom) . At inference time, given an input $x$ , OpenScholar first uses a retriever to identify relevant papers from a specialized datastore ( OpenScholar -Datastore), and then uses a reranker to refine and identify the top $N

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。