[Paper Review] Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

Miao Xiong, Zhiyuan Hu+5 more|arXiv (Cornell University)|Jun 22, 2023

Artificial Intelligence in LawSocial Sciences49 citations

TL;DR

The paper benchmarks non-logit-based confidence elicitation methods for LLMs, comparing verbalized, consistency-based, and hybrid approaches across multiple models and tasks to assess calibration and reliability of expressed uncertainty.

ABSTRACT

Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.

Motivation & Objective

• Motivate the need for confidence elicitation methods that do not rely on model logits or fine-tuning for LLMs, including closed-source APIs.
• Introduce three families of methods—verbalized, consistency-based, and hybrid—and set up a comprehensive benchmarking framework.
• Analyze how these methods perform across diverse reasoning tasks and model sizes to identify strengths, weaknesses, and directions for improvement.
• Provide insights on calibration, reliability, and practical deployment considerations of LLM confidence estimates.

Proposed method

• Define three categories of confidence elicitation methods: verbalized (directly outputs confidence), consistency-based (uses multiple responses to estimate confidence), and hybrid (combines both signals).
• Evaluate four LLMs (GPT-3, GPT-3.5, GPT-4, Vicuna) on five datasets spanning commonsense, arithmetic, symbolic, ethics, and professional knowledge tasks.
• Use zero-tuning, no-access-to-logs/embeddings approaches; rely on prompting strategies like Chain-of-Thought (CoT), Top-K, Multi-step confidence, self-consistency, induced-consistency, and hybridization.
• Adopt calibration and failure-detection metrics (ECE, AUROC, AUPRC-Positive, AUPRC-Negative) to quantify confidence quality across tasks.
• Construct hybrid confidence as an aggregation of verbalized and consistency signals to improve reliability over either signal alone.
• Detail prompts and settings for verbalized prompts (vanilla, CoT-based, multi-step, Top-K) and consistency prompts (self-consistency, induced-consistency), plus prompts for hybrid methods.

Experimental results

Research questions

• RQ1 How well can LLMs express their uncertainty without access to logits or fine-tuning?
• RQ2 Do consistency-based approaches outperform vanilla verbalized confidence across diverse tasks and models?
• RQ3 Can hybrid approaches combining verbalized and consistency signals yield superior calibration and failure-detection performance?
• RQ4 What are the task- and model-dependent strengths and limitations of each confidence elicitation approach, especially on professional knowledge tasks?

Key findings

• Verbalized confidences in vanilla form are highly overconfident across models and tasks.
• Prompting strategies like CoT, Top-K, and Multi-step improve calibration of verbalized confidence, with Top-K often most effective.
• Consistency-based methods generally outperform verbalized methods, notably on arithmetic reasoning tasks.
• Hybrid methods achieve state-of-the-art performance in many cases, outperforming baselines on 13 of 20 cases and improving failure-detection metrics.
• All methods still struggle on challenging tasks requiring professional knowledge, indicating room for further improvement.