[Paper Review] Hallucination is Inevitable: An Innate Limitation of Large Language Models

Ziwei Xu, Sanjay Jain+1 more|arXiv (Cornell University)|Jan 22, 2024

Ferroelectric and Negative Capacitance DevicesEngineering151 citations

TL;DR

The paper formalizes hallucination in a computable world and proves that hallucination is inevitable for any computable LLM, supported by theoretical diagonalization arguments and an empirical validation on real-world tasks, with discussion on mitigators and deployment implications.

ABSTRACT

Hallucination has been widely recognized to be a significant drawback for large language models (LLMs). There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

Motivation & Objective

• Define hallucination in a formal world of computable functions for LLMs.
• Prove that hallucination is unavoidable for any computable LLM under broad conditions.
• Connect formal results to real-world LLM limitations and practical mitigation/deployment implications.

Proposed method

• Define the alphabet, strings, and LLM as a total computable function completing prompts.
• Introduce a formal ground truth function f and a training sample set T for learning f.
• Apply diagonalization to show that no enumeration of P-proved total computable LLMs can learn all computable f, leading to hallucination.
• Generalize to computably enumerable sets and to all computable LLMs, showing hallucination on infinitely many inputs.
• Discuss implications for the design of hallucination mitigators and safe deployment.

Experimental results

Research questions

• RQ1 Can a computable LLM learn to perfectly reproduce the ground-truth function f for all inputs within the formal world?
• RQ2 Are hallucinations inevitable for all computable LLMs across broad classes and training conditions?
• RQ3 What are the practical implications of inevitable hallucination for real-world deployment and mitigator design?

Key findings

• Hallucination is shown to be inevitable for the set of LLMs provably total computable.
• LLMs will hallucinate on infinitely many inputs for computably enumerable sets of LLMs.
• For any computable LLM, there exists a computable ground-truth function f such that the LLM hallucinates on infinitely many inputs.
• Corollaries link polynomial-time constraints and classic problems (e.g., NP-complete problems) to inevitable hallucination in practice.
• Empirical validation demonstrates hallucination-prone behavior on selected real-world problems, supporting the formal results.
• The work discusses how results affect the evaluation, mitigation strategies, and deployment safety of LLMs.