[Paper Review] Estimating the Hallucination Rate of Generative AI

Andrew Jesson, Nicolas Beltran-Velez+6 more|arXiv (Cornell University)|Jun 11, 2024

Mental Health Research Topics4 citations

TL;DR

This paper proposes the Posterior Hallucination Rate (PHR), a Bayesian method to estimate the probability that a generative AI model will produce hallucinated outputs in in-context learning (ICL) settings. By sampling from the model's predictive distribution and evaluating response log probabilities, PHR provides an accurate, uncertainty-based estimate of hallucination risk without requiring external classifiers or task-specific adjustments, validated across synthetic and natural language tasks using Llama-2 models.

ABSTRACT

This paper presents a method for estimating the hallucination rate for in-context learning (ICL) with generative AI. In ICL, a conditional generative model (CGM) is prompted with a dataset and a prediction question and asked to generate a response. One interpretation of ICL assumes that the CGM computes the posterior predictive of an unknown Bayesian model, which implicitly defines a joint distribution over observable datasets and latent mechanisms. This joint distribution factorizes into two components: the model prior over mechanisms and the model likelihood of datasets given a mechanism. With this perspective, we define a hallucination as a generated response to the prediction question with low model likelihood given the mechanism. We develop a new method that takes an ICL problem and estimates the probability that a CGM will generate a hallucination. Our method only requires generating prediction questions and responses from the CGM and evaluating its response log probability. We empirically evaluate our method using large language models for synthetic regression and natural language ICL tasks.

Motivation & Objective

• To develop a method for estimating the hallucination rate in in-context learning (ICL) with conditional generative models (CGMs).
• To define hallucinations as predictions with low probability under the true latent parameter, grounded in a Bayesian interpretation of ICL.
• To create a computationally efficient estimator that only requires model generation and log probability evaluation.
• To validate the method across synthetic regression tasks and natural language ICL benchmarks using Llama-2 models.
• To demonstrate that the PHR estimator correlates strongly with empirical error rates and captures epistemic uncertainty.

Proposed method

• Formalize ICL as Bayesian inference, where the CGM samples from a posterior predictive distribution over an unknown latent model.
• Define the Posterior Hallucination Rate (PHR) as the probability that a generated prediction has low likelihood under the true latent parameter.
• Develop a finite-sample estimator that uses Monte Carlo sampling of context completions and evaluates the log probability of generated responses.
• Use the predictive distribution of the CGM to estimate PHR without requiring access to the true model or external classifiers.
• Validate the estimator using synthetic data and natural language tasks, comparing PHR to empirical error rates and mutual information.
• Demonstrate that PHR correlates strongly with mutual information, indicating it captures epistemic uncertainty.

Experimental results

Research questions

• RQ1 Can the hallucination rate in in-context learning be estimated using a Bayesian framework that treats the model as performing posterior predictive inference?
• RQ2 How accurately can the proposed PHR estimator predict the true hallucination rate in synthetic regression tasks?
• RQ3 Does the PHR estimator correlate with empirical error rates in natural language ICL tasks using Llama-2 models?
• RQ4 Is the PHR estimator robust to variations in context length, number of few-shot examples, and model size?
• RQ5 How does the PHR relate to other uncertainty measures like mutual information in capturing epistemic uncertainty?

Key findings

• The PHR estimator achieved an R² of 0.711 on the SST-2 benchmark when using 20 Monte Carlo samples and 100 y-samples, indicating strong correlation with empirical error rates.
• In synthetic regression tasks, the PHR estimator accurately predicted the true hallucination rate, with a significant reduction in error as context size increased.
• The PHR showed a more linear relationship with the true hallucination rate than mutual information, suggesting it is a more reliable measure of epistemic uncertainty.
• Mutual information was significantly correlated with both the empirical error rate and the PHR across all evaluated tasks, supporting the PHR as a proxy for uncertainty.
• Increasing the number of Monte Carlo and y-samples improved the PHR estimator’s accuracy, while increasing only the number of generated examples or model size degraded performance.
• The PHR estimator demonstrated strong calibration, with low mean absolute error (MAE) across multiple datasets, including AG News, Medical QP, RTE, and WNLI.