[论文解读] What Large Language Models Know and What People Think They Know

Mark Steyvers, Heliodoro Tejeda+6 more|arXiv (Cornell University)|Jan 24, 2024

Topic Modeling被引用 4

一句话总结

本研究调查了大型语言模型（LLMs）内部置信度与其生成回答在多选题问答任务中人类用户感知置信度之间的校准差距。通过使用定制提示使LLM的解释反映其内部置信水平，研究人员显著提升了用户准确评估模型可靠性能力，减少了过度自信现象，并增强了对AI输出的信任。

ABSTRACT

As artificial intelligence (AI) systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs' internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models' actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models' internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in AI-assisted decision-making environments. Code and Data can be found at https://osf.io/y7pr6/ . Journal publication can be found on Nature Machine Intelligence at https://www.nature.com/articles/s42256-024-00976-7 .

研究动机与目标

• 调查LLM生成回答中其内部置信度与人类用户对该置信度感知之间的差异。
• 检验通过修改解释风格是否可提升人类对LLM输出的置信度。
• 评估模型置信度感知解释对用户区分正确与错误答案能力的影响。
• 评估自我评定专业度是否影响用户判断LLM可靠性能力。
• 开发并测试一种方法，使人类对LLM置信度的感知与模型实际内部置信度对齐。

提出的方法

• 使用MMLU数据集开展行为实验，向参与者呈现多选题及其对应的LLM回答。
• 收集用户对每个LLM回答的置信度估计（以概率形式），反映用户对答案准确性的主观判断。
• 采用两种不同的提示策略生成LLM解释：默认解释与基于内部模型置信度的置信度修改解释。
• 使用置信度修改提示，生成从低置信到高置信的解释（例如，'我不太确定……' 对比 '我很有把握……'），以反映模型的内部确定性。
• 比较不同解释类型及LLM（GPT-3.5与PaLM2）下用户在准确度估计与区分能力（AUC）方面的表现。
• 分析自我评定专业度对用户置信度校准及区分能力表现的影响。

Figure 1 : Overview of the evaluation methodology for assessing the calibration gap between model confidence and human confidence in the model. The approach works as follows: (1) prompt the LLM with a multiple-choice question to obtain the model’s internal confidence for each answer choice; (2) sele

实验结果

研究问题

• RQ1 LLM内部置信度与人类对LLM可靠性的感知之间，校准差距有多大？
• RQ2 当解释被调整以反映LLM内部置信度时，人类对LLM输出的置信度质量是否得到提升？
• RQ3 置信度修改的解释是否能减少用户对LLM生成答案的过度自信？
• RQ4 自我评定专业度是否影响用户准确判断LLM表现的能力？
• RQ5 在用户置信度校准方面，不同LLM（如GPT-3.5与PaLM2）之间有何差异？

主要发现

• 当依赖默认解释时，用户普遍高估了LLM回答的准确性，尤其在错误答案上表现明显。
• 当解释被调整以反映LLM的内部置信度时，用户在置信度估计上的校准显著改善，更贴近模型的实际置信水平。
• 使用置信度修改解释后，用户区分正确与错误答案的AUC值提升，表明其区分能力增强。
• 该方法在GPT-3.5与PaLM2上均表现出一致的用户置信度校准改进，证明了方法的稳健性。
• 自我评定专业度并未显著影响用户对LLM准确性的估计能力或对正确与错误回答的区分能力。
• 在默认设置下，校准差距显著，用户对错误答案赋予的置信度高于模型置信度所应支持的水平。

(a) Phase 1