[Paper Review] Do Language Models Know When They're Hallucinating References?

Ayush Agrawal, Suzgun, Mirac+2 more|arXiv (Cornell University)|May 29, 2023

Topic Modeling23 citations

TL;DR

The paper studies open-domain hallucinated references in LMs, introduces direct and indirect black-box queries to detect hallucinations without external data, and shows ensemble approaches improve grounding accuracy across models. It also provides a quantified evaluation using search engine ground truth to compare models and detection methods.

ABSTRACT

State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.

Motivation & Objective

• Assess the prevalence of hallucinated references across state-of-the-art LMs.
• Propose black-box direct and indirect querying methods to judge whether a generated reference is grounded or hallucinated without external resources.
• Quantify detection performance and analyze trade-offs (ROC, FDR) across model sizes.
• Explore whether generation-time changes can mitigate hallucinations and provide guidance for reducing them.

Proposed method

• Generate candidate reference titles from topics using the LM at temperature 1 to create 5 references per topic.
• Label ground-truth grounding of each candidate title via Bing search results as grounded or hallucinated.
• Apply direct queries (three templates) to estimate grounding probability by counting affirmative completions across j samples.
• Apply indirect queries (i>1 responses) to obtain multiple open-ended answers and measure consistency via overlap estimates.
• Compute ROC curves and FDR, and form ensembles (IQ, DQ, IQ+DQ) to improve grounding classification accuracy.

Figure 1 : Example direct vs. indirect LM queries for predicting whether a given paper title is hallucinated or grounded. Direct queries are binary, repeated multiple times to estimate a probability. Indirect queries are open-ended, and their answers are compared to one another, using the LM, to out

Experimental results

Research questions

• RQ1 Can an LM itself reliably detect whether a generated reference is grounded or hallucinated without external data?
• RQ2 Which querying strategy (direct, indirect, or ensemble) yields the best grounding detection across different model sizes?
• RQ3 How do detection methods trade off false discoveries against preserving grounded references?
• RQ4 Do generation-time adjustments (and model size) impact the prevalence of hallucinated references?

Key findings

• Newer models show higher grounding rates (lower hallucination) but still produce substantial hallucinations across GPT-4, ChatGPT, and GPT-3.
• Direct and indirect querying methods can classify hallucinations with measurable AUC, improving with model size (GPT-4 best with IQ and DQ).
• Indirect queries generally outperform direct ones for GPT-3 and ChatGPT, while GPT-4 benefits from both approaches; ensemble IQ+DQ provides the best performance across models.
• Ground-truth labeling via web search (Bing) is a practical proxy for groundedness and supports evaluation of hallucination rates.
• An ensemble of approaches (IQ+DQ) yields higher accuracy than any single method, with trade-offs between FDR and references preserved.

Figure 2 : The prompt used to generate $k=5$ reference titles. This method generates both grounded and hallucinated references. Topics are chosen from the ACM Computing Classification System.