[论文解读] FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

Tu Vu, Mohit Iyyer+9 more|arXiv (Cornell University)|Oct 5, 2023

Topic Modeling被引用 6

一句话总结

本文提出FreshQA，一个动态演化的问答基准，用于测试大语言模型（LLM）在处理快速变化的世界知识和错误前提方面的能力。该研究提出FreshPrompt，一种少样本上下文学习方法，通过检索最新搜索结果来增强LLM的事实性，相较于原始GPT-4，在严格事实性评估下实现了49.0%的准确率提升。

ABSTRACT

Most large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.

研究动机与目标

• 评估大语言模型（LLMs）在实时、动态世界知识场景下的事实性。
• 识别LLMs在快速变化知识和错误前提方面存在的持续性局限。
• 开发一种方法，使LLMs能够在推理时通过搜索引擎动态更新其知识。
• 创建一个可持续、定期更新的基准，以支持动态LLM评估的持续研究。

提出的方法

• FreshQA被构建为一个动态的、人工标注的基准，包含600个开放式问题，分为四类：恒定不变、缓慢变化、快速变化和错误前提。
• 采用双模式评估协议：'宽松'模式仅衡量主答案的正确性，而'严格'模式评估响应中所有陈述的事实准确性，以检测幻觉现象。
• FreshPrompt是一种少样本上下文学习方法，从Google搜索中检索相关且最新的证据，并将其整合到提示中以引导推理。
• 该方法利用检索到的搜索结果——包括相关问题和答案框——作为少样本示例，指导LLM基于最新证据进行推理。
• 检索到的证据数量及其顺序被证明显著影响答案的正确性，最优性能在中等检索长度下实现。
• 该方法无需训练，且完全兼容现有LLMs，可在不重新训练的情况下实现实时事实性提升。

Figure 1: FreshQA exemplars. Our questions are broadly divided into four main categories based on the nature of the answer: never-changing , in which the answer almost never changes; slow-changing , in which the answer typically changes over the course of several years; fast-changing , in which the

实验结果

研究问题

• RQ1 当前LLMs在需要最新世界知识的问题上表现如何，特别是涉及快速变化或错误前提的问题？
• RQ2 模型规模在动态知识基准（如FreshQA）上的提升程度如何？
• RQ3 搜索引擎增强的提示能否显著减少LLM生成响应中的幻觉？
• RQ4 检索证据的数量和顺序等因子如何影响LLM的推理和事实准确性？
• RQ5 简单的少样本提示方法能否优于更复杂的检索增强生成或商业系统？

主要发现

• 所有LLMs，无论规模大小，在快速变化和错误前提问题上均表现出显著性能下降，表明其在动态知识适应方面存在系统性局限。
• 原始GPT-4在严格事实性评估模式下于FreshQA上的准确率仅为51.0%，凸显了广泛存在的幻觉现象。
• FreshPrompt在宽松设置下使GPT-4的准确率提升32.6%，在严格设置下提升49.0%，显著优于Self-Ask和Perplexity.AI。
• 检索证据的数量和顺序至关重要：在中等数量的高质量结果下可实现最优性能，且合理排序可提升推理能力。
• 鼓励简洁直接的回答可减少幻觉，即使在有事实证据的情况下，也优于冗长的回答。
• FreshQA为动态维护和更新，随着世界事件演变，答案会随时间重新分类，确保了基准的长期相关性。

Figure 2: Accuracy of different LLMs on FreshQA under Relaxed and Strict (no hallucination) evaluations. Models benchmarked on the same date of April 26, 2023 . All models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises .