[Paper Review] Comparing Hallucination Detection Metrics for Multilingual Generation

Haoqiang Kang, Terra Blevins+1 more|arXiv (Cornell University)|Feb 16, 2024

Mental Health Research Topics4 citations

TL;DR

This paper evaluates the effectiveness of automatic hallucination detection metrics—lexical (ROUGE, named entity overlap) and NLI-based—across multiple languages in biographical text generation. It finds that NLI-based metrics perform well for sentence-level hallucinations in high-resource languages but fail on atomic facts, while lexical metrics show poor correlation with NLI metrics, highlighting a critical gap in multilingual hallucination detection that demands more robust, language-agnostic methods.

ABSTRACT

While many hallucination detection techniques have been evaluated on English text, their effectiveness in multilingual contexts remains unknown. This paper assesses how well various factual hallucination detection metrics (lexical metrics like ROUGE and Named Entity Overlap, and Natural Language Inference (NLI)-based metrics) identify hallucinations in generated biographical summaries across languages. We compare how well automatic metrics correlate to each other and whether they agree with human judgments of factuality. Our analysis reveals that while the lexical metrics are ineffective, NLI-based metrics perform well, correlating with human annotations in many settings and often outperforming supervised models. However, NLI metrics are still limited, as they do not detect single-fact hallucinations well and fail for lower-resource languages. Therefore, our findings highlight the gaps in exisiting hallucination detection methods for non-English languages and motivate future research to develop more robust multilingual detection methods for LLM hallucinations.

Motivation & Objective

• To assess the effectiveness of existing hallucination detection metrics—originally developed for English—across multilingual text generation.
• To investigate whether lexical metrics (e.g., ROUGE, named entity overlap) and NLI-based metrics correlate in detecting factual hallucinations in non-English languages.
• To evaluate the performance of reference-based versus pairwise metrics in detecting hallucinations across diverse languages.
• To examine the impact of language resource availability on hallucination detection accuracy, especially in low-resource settings.
• To identify limitations of current automatic metrics in detecting atomic fact hallucinations across multilingual contexts.

Proposed method

• The study uses the BLOOMZ-mt model to generate multilingual biographical summaries across 18 languages, including high- and low-resource languages.
• It applies reference-based metrics (e.g., ROUGE, named entity overlap) and NLI-based metrics to compare generated texts against gold-standard references.
• Pairwise metrics are computed by comparing multiple generated samples from the same prompt to assess consistency and detect hallucinations.
• Human evaluation is conducted for Chinese and English to validate automated metrics, using factuality verification as a gold standard.
• Correlation analysis is performed between automated metrics and human judgments to assess reliability across languages.
• The study evaluates performance on both sentence-level hallucinations and simpler atomic fact hallucinations to isolate metric limitations.

Experimental results

Research questions

• RQ1 How well do traditional lexical metrics (ROUGE, named entity overlap) detect factual hallucinations in multilingual text generation compared to NLI-based metrics?
• RQ2 To what extent are NLI-based metrics effective in detecting hallucinations in high-resource versus low-resource languages?
• RQ3 How do reference-based and pairwise NLI-based metrics correlate with human-annotated factuality judgments across languages?
• RQ4 Why do NLI-based metrics fail to detect atomic fact hallucinations despite performing well on sentence-level hallucinations?
• RQ5 To what extent does language resource availability affect the reliability of automatic hallucination detection metrics?

Key findings

• Lexical overlap metrics such as ROUGE and named entity overlap show no significant correlation with NLI-based metrics in detecting factual hallucinations, even in high-resource languages.
• NLI-based metrics perform effectively at detecting sentence-level hallucinations in high-resource languages when compared to human evaluations, but their performance drops significantly for atomic fact hallucinations.
• Pairwise NLI-based metrics strongly correlate with reference-based metrics in high-resource languages, but this correlation weakens substantially in low-resource languages.
• The effectiveness of NLI-based metrics is highly dependent on the performance of underlying NLI models, which are less reliable in low-resource settings, limiting their generalizability.
• Despite their promise, NLI-based metrics fail to detect simple, atomic factual hallucinations, indicating a critical limitation in current factuality verification approaches.
• The study reveals a significant gap in multilingual hallucination detection, particularly for low-resource languages, where simpler metrics like lexical overlap are insufficient and NLI-based methods lack reliability.