[Paper Review] How Language Model Hallucinations Can Snowball

Muru Zhang, Ofir Press+3 more|arXiv (Cornell University)|May 22, 2023

Topic Modeling71 citations

TL;DR

The paper shows that language model hallucinations can snowball: an initial incorrect answer is often followed by an incorrect justification, but models can recognize those snowballed errors when queried separately. It provides three QA datasets and analyzes detection/mitigation strategies for ChatGPT and GPT-4.

ABSTRACT

A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements. Hallucinations are often attributed to knowledge gaps in LMs, but we hypothesize that in some cases, when justifying previously generated hallucinations, LMs output false claims that they can separately recognize as incorrect. We construct three question-answering datasets where ChatGPT and GPT-4 often state an incorrect answer and offer an explanation with at least one incorrect claim. Crucially, we find that ChatGPT and GPT-4 can identify 67% and 87% of their own mistakes, respectively. We refer to this phenomenon as hallucination snowballing: an LM over-commits to early mistakes, leading to more mistakes that it otherwise would not make.

Motivation & Objective

• Motivate and characterize hallucination snowballing as a failure mode in LMs.
• Empirically demonstrate the prevalence of snowballing using three QA datasets.
• Quantify how often state-of-the-art models recognize their own snowballed errors in isolation.

Proposed method

• Automatically construct three yes/no QA datasets (primality, senator alma mater, graph connectivity) where incorrect answers lead to verifiable incorrect claims in explanations.
• Evaluate ChatGPT (gpt-3.5-turbo) and GPT-4 with greedy decoding on zero-shot prompts.
• Extract and verify the model’s incorrect claims and test whether the model can recognize these claims in a separate session.
• Assess the impact of prompting (e.g., “Let’s think step-by-step”) and decoding strategies (temperature, top-k, nucleus, beam search) on snowballing.
• Provide datasets and code publicly to enable replication and further study.

Experimental results

Research questions

• RQ1 Do LMs frequently produce incorrect answers that are accompanied by incorrect, yet testable, justifications?
• RQ2 Can LMs identify and verify their own snowballed hallucinations when prompted to check the incorrect claims in isolation?
• RQ3 How effective are prompting and decoding strategies in reducing snowball hallucinations?
• RQ4 What are the limitations of current models in avoiding snowballing during reasoning tasks?

Key findings

• ChatGPT and GPT-4 exhibit low overall QA accuracy across the three datasets (average accuracy: ChatGPT ~39.87%, GPT-4 ~16.6%).
• Both models commit to an answer within the first token (Yes/No) in over 95% of cases, and these initial commits are frequently incorrect.
• ChatGPT recognizes 67.37% of its snowballed incorrect claims; GPT-4 recognizes 87.03% of such claims when evaluated in isolated verification.
• Prompting with step-by-step reasoning improves task accuracy on some datasets (e.g., Senator Search) but can introduce reasoning errors and still leave snowballing at high levels.
• Higher-temperature decoding and sampling methods do not eliminate snowballing; beam search—unavailable in OpenAI API—could potentially mitigate it, while backtracking prompts may help in some cases.