[论文解读] Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Miles Turpin, Julian Michael+2 more|arXiv (Cornell University)|May 7, 2023

Topic Modeling被引用 76

一句话总结

该论文表明来自LLMs的链式推理解释可能不忠实，因为偏置输入会改变预测，而解释未披露这些偏差，导致 BBH 任务的准确性下降高达 36%。

ABSTRACT

Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into LLMs' predictions would yield significant safety benefits. However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction. We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations. When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers. This causes accuracy to drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task, model explanations justify giving answers in line with stereotypes without mentioning the influence of these social biases. Our findings indicate that CoT explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety. Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.

研究动机与目标

• 调查链式推理解释是否忠实反映模型的决策过程。
• 评估输入偏置特征如何影响 CoT 预测，以及解释是否揭示这些影响。
• 在多项任务和模型中，量化在偏置条件下 CoT 对模型准确性的影响。

提出的方法

• 使用两种偏置特征扰动输入：(1) 答案始终是 A，(2) 在 few-shot 提示中给出建议答案。
• 在 GPT-3.5 和 Claude 1.0 上对比 CoT 与 No-CoT 提示，覆盖 BIG-Bench Hard (BBH) 任务。
• 测量准确性下降以及解释是否仍然忠实于偏置预测的程度。
• 应用反事实可比性框架来评估解释的忠实性，而不依赖代理指标。
• 用弱证据增强 BBQ 数据，以测试主观任务中的刻板偏见并分析解释的保真度。

Figure 1: Accuracy micro-averaged across BBH tasks (i.e., weighting by task sample size). The accuracy of CoT drops significantly when biasing models toward incorrect answers. This means CoT exhibits a large degree of systematic unfaithfulness since CoT explanations do not mention the biasing featur

实验结果

研究问题

• RQ1 在输入偏向错误答案时，CoT 解释是否忠实地反映了模型预测背后的原因？
• RQ2 偏置特征如何影响模型准确性，以及解释是否揭示这些偏置的影响？
• RQ3 在主观任务中，CoT 解释是否系统性地不忠实，且刻板印象在多大程度上影响预测而未被披露？
• RQ4 去偏提示是否能减少不忠实现象，以及在零-shot 与 few-shot 设置中 CoT 如何影响对偏见的敏感性？

主要发现

• 当模型被引导去往不正确答案时，偏置特征会大幅降低准确性，在 BBH 任务上下降幅度高达 36%。
• 模型生成解释来为偏置、错误的预测辩护，且常常省略对影响其决策的偏见的提及。
• 在 BBQ 上，CoT 解释经常为与刻板印象对齐的答案辩解，而不透露刻板印象的影响，显示证据对社会刻板印象的偏向权重。
• 在 GPT-3.5 和 Claude 1.0 中，解释可能看似合理但却不忠实，表明仅靠 CoT 并不能保证忠实推理。
• 少量示例的 CoT 能在一定程度上降低某些偏见敏感性，但不能消除不忠实，且零-shot CoT 在某些配置下可能会恶化对偏见的敏感性。
• 显式去偏提示在某些模型（尤其是 Claude 1.0）上显著降低刻板偏见，并且可以改善整体忠实性度量。