[Paper Review] HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models

Junyi Li, Xiaoxue Cheng+3 more|arXiv (Cornell University)|May 19, 2023

Mental Health via WritingPsychology29 citations

TL;DR

HaluEval provides a large-scale benchmark (30,000 hallucinated samples and 5,000 annotated ChatGPT responses) to study LLM hallucinations and their recognition by various models and strategies.

ABSTRACT

Large language models (LLMs), such as ChatGPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval), a large collection of generated and human-annotated hallucinated samples for evaluating the performance of LLMs in recognizing hallucination. To generate these samples, we propose a ChatGPT-based two-step framework, i.e., sampling-then-filtering. Besides, we also hire some human labelers to annotate the hallucinations in ChatGPT responses. The empirical results suggest that ChatGPT is likely to generate hallucinated content in specific topics by fabricating unverifiable information (i.e., about $19.5\%$ responses). Moreover, existing LLMs face great challenges in recognizing the hallucinations in texts. However, our experiments also prove that providing external knowledge or adding reasoning steps can help LLMs recognize hallucinations. Our benchmark can be accessed at https://github.com/RUCAIBox/HaluEval.

Motivation & Objective

• Understand what types of content LLMs tend to hallucinate and to what extent they do so.
• Create a scalable two-stage sampling-then-filtering pipeline to automatically generate hallucinated samples.
• Provide human annotations to identify and locate hallucinated spans in ChatGPT responses.
• Evaluate multiple LLMs’ ability to recognize hallucinations across tasks (QA, knowledge-grounded dialogue, summarization).
• Propose strategies (knowledge retrieval, reasoning) to improve hallucination recognition.

Proposed method

• Two-stage sampling-then-filtering pipeline to generate hallucinated samples auto­matically.
• Two sampling styles for general queries: one-pass and conversational instruction schemas.
• Automatic generation of 30,000 task-specific hallucinated samples (QA, dialogue, summarization) from seed datasets and filtering with ground-truth cues.
• Human annotation of 5,000 ChatGPT responses for hallucinations with span-level labeling.
• Evaluation of multiple LLMs (closed- and open-source) on their ability to recognize hallucinations.
• Analysis of improvement strategies such as external knowledge retrieval and chain-of-thought reasoning.

Experimental results

Research questions

• RQ1 What types of content do LLMs hallucinate, and how prevalent are such hallucinations across tasks?
• RQ2 How well can current LLMs recognize hallucinations in generated text?
• RQ3 Do external knowledge, reasoning steps, or contrastive setups improve hallucination recognition?
• RQ4 Are hallucinations topic-sensitive, and which topics pose the most challenges for recognition?

Key findings

• ChatGPT-generated hallucinations occur in approximately 19.5% of general responses.
• LLMs show substantial difficulty recognizing hallucinations, e.g., ChatGPT achieves 58.53% accuracy on summarization and 62.59% in QA-language tasks, with some models near random chance.
• Providing external knowledge significantly improves recognition (e.g., QA accuracy rises to 76.83% with knowledge retrieval).
• Chain-of-thought reasoning offers mixed results: it helps in summarization but can reduce performance in QA and dialogue.
• Ground-truth contrastive testing is challenging and can decrease recognition performance, highlighting benchmark difficulty.
• Topic analysis indicates hallucination recognition failures concentrate on certain topics (e.g., film, technology, language) and vary by task.