[Paper Review] Cycles of Thought: Measuring LLM Confidence through Stable Explanations

Evan Becker, Stefano Soatto|arXiv (Cornell University)|Jun 5, 2024

Auction Theory and ApplicationsDecision Sciences3 citations

TL;DR

This paper proposes a novel framework for measuring large language model (LLM) uncertainty by analyzing the stability of generated explanations, treating each explanation-answer pair as a test-time classifier. By computing a posterior distribution over these classifiers using explanation entailment, the method achieves state-of-the-art performance in selective uncertainty (AUROC and AURC) across five diverse benchmarks, particularly improving on complex reasoning tasks.

ABSTRACT

In many high-risk machine learning applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer. While utilizing explanations is not a new idea in and of itself, by interpreting each possible model+explanation pair as a test-time classifier we can calculate a posterior answer distribution over the most likely of these classifiers. We demonstrate how a specific instance of this framework using explanation entailment as our classifier likelihood improves confidence score metrics (in particular AURC and AUROC) over baselines across five different datasets. We believe these results indicate that our framework is both a well-principled and effective way of quantifying uncertainty in LLMs.

Motivation & Objective

• To address the critical problem of LLM overconfidence in incorrect predictions, especially in high-stakes applications.
• To develop a black-box uncertainty quantification method that does not require access to model weights or internal states.
• To improve selective uncertainty performance—distinguishing correct from incorrect predictions—beyond existing heuristic methods.
• To formalize uncertainty estimation through transductive marginalization over explanation-based classifiers.
• To evaluate the method across diverse, complex benchmarks to assess robustness and generalization.

Proposed method

• The method treats each LLM-generated explanation-answer pair as a test-time classifier, forming a distribution over possible classifiers.
• It computes a posterior predictive distribution by marginalizing over these classifiers, using explanation entailment as a proxy for classifier likelihood.
• Explanation entailment is used to filter and reweight explanations based on logical consistency with the answer, improving confidence calibration.
• The framework leverages chain-of-thought reasoning at test time, but focuses on the stability of explanations rather than just their content.
• The final confidence score is derived from the distribution of stable, entailment-verified explanation-answer pairs.
• The approach is implemented using GPT-3.5 and GPT-4, with evaluation on five diverse datasets using standard uncertainty metrics.

Figure 1 : Empirical distribution of explanation log likelihoods (top left) and explanation entailment probabilities (top right) generated for the TruthQA dataset using token logits from GPT3.5-Turbo. Red denotes explanations generated by conditioning on the incorrect answer and blue denotes explana

Experimental results

Research questions

• RQ1 Can explanation stability be used as a reliable proxy for LLM confidence in the absence of model internals?
• RQ2 Does using entailment to weight explanation-based classifiers improve selective uncertainty performance compared to baselines?
• RQ3 How does the method perform across datasets of varying complexity, particularly on reasoning-intensive tasks?
• RQ4 To what extent does the method reduce overconfidence in incorrect answers without relying on heuristic self-verbalization?
• RQ5 Can test-time explanation distributions be used to approximate a Bayesian posterior predictive distribution in LLMs?

Key findings

• The proposed stable explanations confidence method outperforms all baselines in selective uncertainty, achieving the highest AUROC and AURC scores on average across five datasets.
• On the MMLU Professional Law dataset, the method achieved an AUROC of 0.784 and AURC of 0.131 with GPT-3.5-Turbo, significantly outperforming the next best method.
• For GPT-4-Turbo, the method achieved an AUROC of 0.899 and AURC of 0.131 on CommonsenseQA, demonstrating strong performance on complex reasoning tasks.
• The ablation study confirmed that combining entailment reweighting and explanation-conditioned answer distributions yields the best performance, outperforming individual components.
• The method showed particular gains on complex, multi-step reasoning questions (e.g., MMLU Law and Physics), where baseline methods often fail.
• Despite strong gains in selective uncertainty, calibration (ECE) performance was not state-of-the-art, which the authors attribute to the fundamental limitations of ECE under distribution shift.

Figure 3 : Stable Explanation Prompts