[论文解读] Do Language Models Know When They're Hallucinating References?

Ayush Agrawal, Suzgun, Mirac+2 more|arXiv (Cornell University)|May 29, 2023

Topic Modeling被引用 23

一句话总结

本论文研究语言模型中的开放领域幻觉引用，提出直接和间接的黑箱查询以在无外部数据的情况下检测幻觉，并显示集成方法在不同模型中的 grounding 准确性有所提升。它还提供了使用搜索引擎真实数据进行量化评估以比较模型和检测方法。

ABSTRACT

State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.

研究动机与目标

• 评估最先进语言模型中幻觉引用的普遍性。
• 提出黑箱直接与间接查询方法，以在无外部资源的情况下判断生成的引用是否有据可依。
• 量化检测性能并分析取舍（ROC、FDR），覆盖不同模型规模。
• 探索生成时的调整是否能减轻幻觉，并为降低幻觉提供指导。

提出的方法

• 使用温度为1的语言模型从主题中生成候选参考标题，每个主题生成5个引用。
• 通过Bing搜索结果对每个候选标题的真实有据性进行标注，标注为有据或幻觉。
• 应用直接查询（三个模板）通过统计在 j 次采样中肯定完成的数量来估计有据概率。
• 应用间接查询（i>1 的回答）以获得多次开放式回答，并通过重叠估计衡量一致性。
• 计算ROC曲线和FDR，并形成集合（IQ、DQ、IQ+DQ）以提高有据分类的准确性。

Figure 1 : Example direct vs. indirect LM queries for predicting whether a given paper title is hallucinated or grounded. Direct queries are binary, repeated multiple times to estimate a probability. Indirect queries are open-ended, and their answers are compared to one another, using the LM, to out

实验结果

研究问题

• RQ1 语言模型本身是否在没有外部数据的情况下就能可靠地检测生成的引用是否有据可依？
• RQ2 哪种查询策略（直接、间接，或集成）在不同模型规模上能给出最佳的有据检测？
• RQ3 在保持有据引用的同时，检测方法如何权衡误发现（false discoveries）？
• RQ4 生成时的调整（以及模型规模）是否会影响幻觉引用的高发？

主要发现

模型	谵妄率 (%) (H)
GPT-4	46.8%
ChatGPT	59.6%
GPT-3	73.6%

• 较新的模型显示出更高的有据率（幻觉较低），但在GPT-4、ChatGPT和GPT-3等模型中仍会产生相当数量的幻觉。
• 直接和间接查询方法能够以可测量的AUC对幻觉进行分类，并随着模型规模的增大而提高（GPT-4在IQ和DQ上的表现最佳）。
• 相较于直接方法，间接查询通常在GPT-3和ChatGPT上表现更好，而GPT-4同时受益于两种方法；IQ+DQ 的集合在所有模型中提供最佳性能。
• 通过网络搜索（Bing）的真实标注是有据性的实际代理，并支持对幻觉率的评估。
• 方法组合（IQ+DQ）的准确性高于任何单一方法，在FDR与保留引用之间存在权衡。

Figure 2 : The prompt used to generate $k=5$ reference titles. This method generates both grounded and hallucinated references. Topics are chosen from the ACM Computing Classification System.