[Paper Review] Language Models (Mostly) Know What They Know

Saurav Kadavath, Tom Conerly+34 more|arXiv (Cornell University)|Jul 11, 2022

Topic ModelingComputer Science159 citations

TL;DR

The paper shows that large language models are well-calibrated on diverse multiple-choice and true/false tasks when formatted properly, and explores how models can self-evaluate and predict whether they know the answer (P(IK)) without relying on a specific proposed answer.

ABSTRACT

We study whether language models can evaluate the validity of their own claims and predict which questions they will be able to answer correctly. We first show that larger models are well-calibrated on diverse multiple choice and true/false questions when they are provided in the right format. Thus we can approach self-evaluation on open-ended sampling tasks by asking models to first propose answers, and then to evaluate the probability "P(True)" that their answers are correct. We find encouraging performance, calibration, and scaling for P(True) on a diverse array of tasks. Performance at self-evaluation further improves when we allow models to consider many of their own samples before predicting the validity of one specific possibility. Next, we investigate whether models can be trained to predict "P(IK)", the probability that "I know" the answer to a question, without reference to any particular proposed answer. Models perform well at predicting P(IK) and partially generalize across tasks, though they struggle with calibration of P(IK) on new tasks. The predicted P(IK) probabilities also increase appropriately in the presence of relevant source materials in the context, and in the presence of hints towards the solution of mathematical word problems. We hope these observations lay the groundwork for training more honest models, and for investigating how honesty generalizes to cases where models are trained on objectives other than the imitation of human writing.

Motivation & Objective

• Assess whether large language models are calibrated on diverse MCQ, True/False, and related tasks when formatted with explicit options.
• Investigate self-evaluation by having models generate and then assess their own outputs.
• Train models to predict the probability that they know the answer (P(IK)) independent of proposed answers.
• Examine generalization of P(IK) across tasks and with the presence of source materials or hints.

Proposed method

• Evaluate 800M, 3B, 12B, and 52B models on BIG Bench, MMLU, TruthfulQA, QuALITY, and LogiQA under various formats.
• Format MCQs with lettered choices and assess calibration via Expected Calibration Error (ECE) and related metrics.
• Test True/False reformulations to measure P(True) calibration.
• Train a value-head to predict P(IK) and compare with a natural-language approach.
• Use self-generated samples (T=1) and self-evaluation prompts to measure P(True) accuracy and Brier scores.

Experimental results

Research questions

• RQ1 Can large language models produce calibrated probabilities for their outputs across diverse tasks when questions are presented with explicit options?
• RQ2 Can models effectively self-evaluate the correctness of their own samples (P(True)) and improve this evaluation by brainstorming multiple samples?
• RQ3 Can models be trained to predict the probability they know the answer (P(IK)) independent of a proposed answer, and how well does this generalize across tasks?
• RQ4 How do source materials or hints affect P(IK) predictions and calibration?
• RQ5 What is the impact of RLHF and prompting formats on model calibration and honesty?

Key findings

• Large models show strong calibration on multiple-choice tasks when options are visible and formatting is favorable; calibration improves with model size and few-shot prompting.
• Replacing an option with ‘none of the above’ degrades performance and calibration, suggesting models struggle with undefined truth when forced to abstain.
• True/False formulations yield well-calibrated predictions (P(True)) across tasks, with calibration robust for larger models.
• RLHF policy calibration can be remedied by simple temperature tuning, improving alignment of predictions.
• Self-evaluation of model-generated samples (P(True)) is feasible and more accurate when models are shown many samples (brainstorming) before verdict; calibration improves with model size.
• Models can be trained to predict P(IK) with a value head, showing cross-task generalization, though calibration is better within-distribution than out-of-distribution.
• P(IK) increases with the availability of source materials and hints for problem-solving, indicating sensitivity to extra context.