[论文解读] Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models

Yuji Zhang, Sha Li+6 more|arXiv (Cornell University)|Jul 10, 2024

Machine Learning in Healthcare被引用 5

一句话总结

论文识别一种被称为知识遮蔽的现象：含有多条件的提示会使LLMs忽视某些条件，产生融合的幻觉，并提出推理时检测和自对比解码以缓解。

ABSTRACT

Hallucination is often regarded as a major impediment for using large language models (LLMs), especially for knowledge-intensive tasks. Even when the training corpus consists solely of true statements, language models still generate hallucinations in the form of amalgamations of multiple facts. We coin this phenomenon as ``knowledge overshadowing'': when we query knowledge from a language model with multiple conditions, some conditions overshadow others, leading to hallucinated outputs. This phenomenon partially stems from training data imbalance, which we verify on both pretrained models and fine-tuned models, over a wide range of LM model families and sizes.From a theoretical point of view, knowledge overshadowing can be interpreted as over-generalization of the dominant conditions (patterns). We show that the hallucination rate grows with both the imbalance ratio (between the popular and unpopular condition) and the length of dominant condition description, consistent with our derived generalization bound. Finally, we propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced, along with a training-free self-contrastive decoding method to alleviate hallucination during inference. Our proposed approach showcases up to 82% F1 for hallucination anticipation and 11.2% to 39.4% hallucination control, with different models and datasets.

研究动机与目标

• 研究在训练数据正确的情况下，包含多个条件的提示为何会在LLM中诱发融合幻觉。
• 描述数据不平衡和条件长度如何影响跨模型家族和不同规模的幻觉率。
• 开发推理时检测遮蔽并在不重新训练的情况下缓解幻觉的策略。

提出的方法

• 将知识遮蔽定义为主导条件 p(y|AB) ≈ p(y|A)，从而忽略较不常见的 B。
• 在多项任务和不同模型规模上，实证展示预训练和微调模型中的遮蔽现象。
• 量化不平衡比、条件长度与幻觉率之间的关系。
• 推导一个将遮蔽与模型泛化通过 NTP 损失和 GSNR 联系起来的一般化界限。
• 提出基于 PMI 的遮蔽检测以及用于检测的逃逸惩罚机制（EPM）。
• 引入自对比解码（SCD）以在推理阶段减少支配偏差。

实验结果

研究问题

• RQ1 当提示包含多重条件时，知识遮蔽是否在不同的模型家族和规模中普遍存在？
• RQ2 数据不平衡和条件长度如何影响自回归大语言模型的幻觉率？
• RQ3 推理时的检测与解码技术是否能在不重新训练的情况下预测并缓解因遮蔽引起的幻觉？
• RQ4 哪些理论见解将遮蔽与下一个标记预测的泛化界限联系起来？

主要发现

• 知识遮蔽在多种模型家族和规模中产生融合的幻觉。
• 幻觉率随不平衡比增加而上升，且较大模型显示更高的相对幻觉率。
• 更长的主导条件描述导致更高的幻觉率，对较小模型曲线更陡峭。
• 一种基于 PMI 信号的无需训练的遮蔽检测器在预测数据集上实现最高 82% 的 F1。
• 自对比解码在不同数据集和模型上将幻觉率降低了 11.2% 至 39.4%。
• 一个理论泛化界将遮蔽与 GSNR 及主导条件长度联系起来。