[论文解读] ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence

Kevin Wu, Eric Wu+1 more|arXiv (Cornell University)|Apr 16, 2024

Corporate Finance and Governance被引用 4

一句话总结

本文提出了ClashEval，一种用于量化检索增强生成（RAG）系统中大型语言模型（LLM）内部知识先验与外部检索证据之间张力的框架。通过系统性地对参考文档施加逐渐错误的值，作者发现当LLM的内部先验较弱时，更可能采纳错误的RAG内容；而当先验较强时，会抵制错误信息——揭示了RAG可靠性中的关键权衡，挑战了其固有事实正确性的假设。

ABSTRACT

Retrieval augmented generation (RAG) is frequently used to mitigate hallucinations and provide up-to-date knowledge for large language models (LLMs). However, given that document retrieval is an imprecise task and sometimes results in erroneous or even harmful content being presented in context, this raises the question of how LLMs handle retrieved information: If the provided content is incorrect, does the model know to ignore it, or does it recapitulate the error? Conversely, when the model's initial response is incorrect, does it always know to use the retrieved information to correct itself, or does it insist on its wrong prior response? To answer this, we curate a dataset of over 1200 questions across six domains (e.g., drug dosages, Olympic records, locations) along with content relevant to answering each question. We further apply precise perturbations to the answers in the content that range from subtle to blatant errors. We benchmark six top-performing LLMs, including GPT-4o, on this dataset and find that LLMs are susceptible to adopting incorrect retrieved content, overriding their own correct prior knowledge over 60% of the time. However, the more unrealistic the retrieved content is (i.e. more deviated from truth), the less likely the model is to adopt it. Also, the less confident a model is in its initial response (via measuring token probabilities), the more likely it is to adopt the information in the retrieved content. We exploit this finding and demonstrate simple methods for improving model accuracy where there is conflicting retrieved content. Our results highlight a difficult task and benchmark for LLMs -- namely, their ability to correctly discern when it is wrong in light of correct retrieved content and to reject cases when the provided content is incorrect.

研究动机与目标

• 调查当提供正确外部证据时，RAG系统是否能可靠地纠正LLM的幻觉。
• 检查LLM是否能够检测并拒绝错误或被扰动的检索信息。
• 量化模型内部知识先验对其遵循RAG所提供内容的影响。
• 评估提示策略如何影响先验知识与检索事实之间的平衡。
• 评估该动态在GPT-4以外的多种领域和LLM中的鲁棒性。

提出的方法

• 作者在有和没有参考文档的情况下向GPT-4及其他LLM提出基于事实的问题，测量响应标记的概率。
• 他们通过逐渐错误的值（如数值异常值、类别错配）系统性地扰动参考文档，以模拟现实世界中的检索错误。
• 对于每个扰动级别，他们计算RAG偏好率——即与修改后文档一致的响应比例。
• 他们将RAG偏好率与模型的先验响应概率（在无上下文情况下的正确标记的对数概率）在1200多个问题上进行相关性分析。
• 他们分析提示风格（例如“严格遵循”与“宽松遵循”）对先验-RAG关系强度和方向的影响。
• 他们将分析扩展到GPT-3.5和Mistral-7B，以评估在不同模型和领域中的泛化能力。

Figure 1: A schematic of generating modified documents for each dataset. A question is posed to the LLM with and without a reference document containing information relevant to the query. This document is then perturbed to contain modified information and given as context to the LLM. We then observe

实验结果

研究问题

• RQ1 提供正确的检索信息是否能可靠地修复LLM的幻觉，还是即使有正确的RAG输入，模型仍可能失败？
• RQ2 模型内部先验的强度在多大程度上影响其采纳或拒绝错误检索信息的倾向？
• RQ3 检索内容与模型先验之间的偏差在多大程度上影响采纳检索事实的可能性？
• RQ4 不同的提示指令（例如“严格遵循”）如何调节先验知识与RAG证据之间的平衡？
• RQ5 这些动态在不同领域和LLM架构中是否具有一致性？

主要发现

• 当提供正确的参考文档时，GPT-4纠正了其初始幻觉的94%，证实了在理想条件下RAG的有效性。
• RAG偏好率与模型先验置信度呈负相关：先验概率每增加10%，采纳RAG内容的可能性平均下降2.3%。
• 先验较弱的LLM显著更可能采纳错误的、被扰动的RAG内容，尤其是在偏差较大时。
• 随着扰动与正确事实的偏差增大，模型越来越倾向于回归其内部先验，表明对不合理信息的抵抗。
• 先验置信度与RAG采纳之间的负相关关系在六个不同数据集中均成立，并在GPT-3.5和Mistral-7B中得到复现，证实了其鲁棒性。
• 即使在要求严格遵循RAG内容的提示指令下，模型仍表现出对内部先验的强烈偏好，尤其是在先验高度自信时。

Figure 2: Across six QA datasets using GPT-4, we consistently observe an inverse relationship between the RAG preference rate (y-axis) and two characteristics (x-axes): 1. the model’s prior response probability (lefthand plots), and 2. the amount of deviation from the prior (righthand plots). RAG pr