[Paper Review] AI Hallucinations: A Misnomer Worth Clarifying

Negar Maleki, Balaji Padmanabhan+1 more|arXiv (Cornell University)|Jan 9, 2024

Artificial Intelligence in Healthcare and Education5 citations

TL;DR

The paper conducts a systematic review of how AI hallucination is defined across 14 databases, revealing no universal definition and proposing standardized terminology and taxonomy.

ABSTRACT

As large language models continue to advance in Artificial Intelligence (AI), text generation systems have been shown to suffer from a problematic phenomenon termed often as "hallucination." However, with AI's increasing presence across various domains including medicine, concerns have arisen regarding the use of the term itself. In this study, we conducted a systematic review to identify papers defining "AI hallucination" across fourteen databases. We present and analyze definitions obtained across all databases, categorize them based on their applications, and extract key points within each category. Our results highlight a lack of consistency in how the term is used, but also help identify several alternative terms in the literature. We discuss implications of these and call for a more unified effort to bring consistency to an important contemporary AI issue that can affect multiple domains significantly.

Motivation & Objective

• Identify how the term AI hallucination is defined across diverse domains and databases.
• Assess consistency of definitions and identify common characteristics and alternatives.
• Compile and categorize definitions to inform a unified terminology for AI-generated content errors.
• Provide guidance toward a formal definition and a robust taxonomy for cross-domain use.

Proposed method

• Perform a broad literature search across 14 databases from 2013 to 2023 for papers defining AI hallucination in AI/LLMs.
• Manually review each retrieved paper to extract definitions and context.
• Aggregate 333 definitions and summarize them in an appendix.
• Classify definitions by application domain (e.g., health, legal, translation, summarization).
• Identify alternative terms and discuss implications for terminology standardization.

Experimental results

Research questions

• RQ1 What definitions of AI hallucination exist across different domains and databases?
• RQ2 How consistent are these definitions, and what characteristics do they share or diverge on?
• RQ3 What alternative terms are used, and how can a unified taxonomy be developed to improve clarity?

Key findings

• There is no precise, universally accepted definition of AI hallucination across the literature.
• Definitions vary by application and can be conflicting or context-dependent.
• Table II and Table III summarize alternative terms and key points used to describe AI hallucination across domains.
• The authors compile 333 definitions from 2013–2023 and provide an Appendix with the full set of definitions.
• There is a push in recent literature to replace or rename the term to avoid mental-health connotations and stigma.
• The paper advocates for a unified terminology and a robust, formal definition to improve cross-domain communication and research.