[Paper Review] What Large Language Models Know and What People Think They Know

Mark Steyvers, Heliodoro Tejeda+6 more|arXiv (Cornell University)|Jan 24, 2024

Topic Modeling4 citations

TL;DR

This study investigates the calibration gap between large language models' (LLMs) internal confidence and human users' perception of that confidence in multiple-choice question-answering tasks. By modifying LLM explanations to reflect internal confidence levels—using tailored prompts—researchers significantly improved users' ability to accurately assess model reliability, reducing overconfidence and enhancing trust in AI outputs.

ABSTRACT

As artificial intelligence (AI) systems, particularly large language models (LLMs), become increasingly integrated into decision-making processes, the ability to trust their outputs is crucial. To earn human trust, LLMs must be well calibrated such that they can accurately assess and communicate the likelihood of their predictions being correct. Whereas recent work has focused on LLMs' internal confidence, less is understood about how effectively they convey uncertainty to users. Here we explore the calibration gap, which refers to the difference between human confidence in LLM-generated answers and the models' actual confidence, and the discrimination gap, which reflects how well humans and models can distinguish between correct and incorrect answers. Our experiments with multiple-choice and short-answer questions reveal that users tend to overestimate the accuracy of LLM responses when provided with default explanations. Moreover, longer explanations increased user confidence, even when the extra length did not improve answer accuracy. By adjusting LLM explanations to better reflect the models' internal confidence, both the calibration gap and the discrimination gap narrowed, significantly improving user perception of LLM accuracy. These findings underscore the importance of accurate uncertainty communication and highlight the effect of explanation length in influencing user trust in AI-assisted decision-making environments. Code and Data can be found at https://osf.io/y7pr6/ . Journal publication can be found on Nature Machine Intelligence at https://www.nature.com/articles/s42256-024-00976-7 .

Motivation & Objective

• To investigate the disparity between LLMs' internal confidence and human users' perception of that confidence in LLM-generated responses.
• To examine whether human confidence in LLM outputs can be improved through modifications to explanation style.
• To assess the impact of model confidence-aware explanations on user discrimination between correct and incorrect answers.
• To evaluate whether self-rated expertise influences users' ability to judge LLM reliability.
• To develop and test a method for aligning human perception of LLM confidence with the model's actual internal confidence.

Proposed method

• Conducted behavioral experiments using the MMLU dataset, presenting participants with multiple-choice questions and corresponding LLM responses.
• Collected human confidence estimates (as probabilities) for each LLM response, reflecting users' perceived accuracy of the answer.
• Generated LLM explanations using two distinct prompting strategies: default explanations and confidence-modified explanations based on internal model confidence.
• Used confidence-modified prompts to generate explanations ranging from low to high confidence (e.g., 'I am not sure...' vs. 'I am confident...') to reflect the model’s internal certainty.
• Compared user performance in accuracy estimation and discrimination (AUC) across different explanation types and LLMs (GPT-3.5 and PaLM2).
• Analyzed the effect of self-rated expertise on user confidence calibration and discrimination performance.

Figure 1 : Overview of the evaluation methodology for assessing the calibration gap between model confidence and human confidence in the model. The approach works as follows: (1) prompt the LLM with a multiple-choice question to obtain the model’s internal confidence for each answer choice; (2) sele

Experimental results

Research questions

• RQ1 How large is the calibration gap between LLM internal confidence and human perception of LLM reliability?
• RQ2 Does the quality of human confidence in LLM outputs improve when explanations are tailored to reflect internal model confidence?
• RQ3 Can confidence-modified explanations reduce user overconfidence in LLM-generated answers?
• RQ4 Does self-rated expertise influence users’ ability to accurately judge LLM performance?
• RQ5 How do different LLMs (e.g., GPT-3.5 vs. PaLM2) compare in terms of user confidence calibration?

Key findings

• Users consistently overestimated the accuracy of LLM responses when relying on default explanations, particularly for incorrect answers.
• When explanations were tailored to reflect the LLM’s internal confidence, users showed significantly improved calibration in their confidence estimates, aligning more closely with the model’s actual confidence levels.
• The AUC for discriminating correct from incorrect answers improved when confidence-modified explanations were used, indicating better user discrimination ability.
• The improvement in user confidence calibration was consistent across both GPT-3.5 and PaLM2, demonstrating the robustness of the approach.
• Self-rated expertise did not significantly affect users’ ability to estimate LLM accuracy or discriminate between correct and incorrect responses.
• The calibration gap was substantial in default settings, with users assigning higher confidence to incorrect answers than warranted by model confidence.

(a) Phase 1