[Paper Review] INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection

Chao Chen, Kai Liu+6 more|arXiv (Cornell University)|Feb 6, 2024

Academic Publishing and Open AccessDecision Sciences3 citations

TL;DR

This paper proposes INSIDE, a framework that leverages large language models' internal states to detect hallucinations by measuring semantic consistency via an EigenScore based on embedding covariance eigenvalues and applying test-time feature clipping to reduce overconfident generations. The method achieves state-of-the-art performance on multiple QA benchmarks by directly analyzing dense semantic representations without relying on post-hoc language-level consistency checks.

ABSTRACT

Knowledge hallucination have raised widespread concerns for the security and reliability of deployed LLMs. Previous efforts in detecting hallucinations have been employed at logit-level uncertainty estimation or language-level self-consistency evaluation, where the semantic information is inevitably lost during the token-decoding procedure. Thus, we propose to explore the dense semantic information retained within LLMs' extbf{IN}ternal extbf{S}tates for halluc extbf{I}nation extbf{DE}tection ( extbf{INSIDE}). In particular, a simple yet effective extbf{EigenScore} metric is proposed to better evaluate responses' self-consistency, which exploits the eigenvalues of responses' covariance matrix to measure the semantic consistency/diversity in the dense embedding space. Furthermore, from the perspective of self-consistent hallucination detection, a test time feature clipping approach is explored to truncate extreme activations in the internal states, which reduces overconfident generations and potentially benefits the detection of overconfident hallucinations. Extensive experiments and ablation studies are performed on several popular LLMs and question-answering (QA) benchmarks, showing the effectiveness of our proposal.

Motivation & Objective

• To address the limitations of logit-level and language-level hallucination detection methods that lose semantic information during decoding.
• To explore whether dense semantic information retained in LLM internal states can enable more accurate hallucination detection.
• To develop a method that detects both inconsistent and overconfident hallucinations by analyzing embedding space properties.
• To improve reliability of LLMs in real-world applications by identifying factually incorrect generations without requiring external models.

Proposed method

• Proposes a generalized framework, INSIDE, that uses internal states of LLMs for hallucination detection instead of relying on decoded text or logits.
• Introduces EigenScore, a metric that computes the eigenvalues of the covariance matrix of multiple response embeddings to measure semantic consistency/diversity in the dense embedding space.
• Demonstrates that EigenScore correlates with differential entropy in the embedding space, serving as a proxy for uncertainty.
• Applies a test-time feature clipping technique to truncate extreme activations in the internal states, reducing overconfident generations.
• Uses auto-regressive decoding with multiple generations per prompt to compute EigenScore and detect inconsistencies.
• Employs standard QA benchmarks (e.g., CoQA, LLaMA-7B) for evaluation with AUROC and correctness measures under varying thresholds.

Figure 1: Illustration of our proposed hallucination detection pipeline. During inference time, for a given question, the extreme features in the penultimate layer are truncated and the EigenScore is computed based on the sentence embeddings across multiple responses.

Experimental results

Research questions

• RQ1 Can internal states of LLMs preserve sufficient semantic information to detect hallucinations more effectively than post-hoc language-level methods?
• RQ2 How does the EigenScore metric, derived from embedding covariance eigenvalues, reflect semantic consistency and detect hallucinations?
• RQ3 Can feature clipping during inference reduce overconfident hallucinations that are otherwise missed by consistency-based methods?
• RQ4 How robust is the proposed method across different LLMs and hyperparameters like temperature and top-k?

Key findings

• EigenScore outperforms baseline methods such as perplexity, LN-entropy, and lexical similarity across all correctness thresholds on CoQA and LLaMA-7B, achieving an AUROC of 80.4 at a 0.9 similarity threshold.
• The method shows strong performance across diverse models and benchmarks, indicating broad applicability beyond specific architectures.
• Feature clipping significantly improves detection of overconfident hallucinations by reducing extreme activations, especially under high temperature settings.
• Sensitivity analysis reveals that performance is highly sensitive to temperature but less so to top-k, with optimal performance in the 0.1–1.0 range.
• The EigenScore metric effectively captures differential entropy in the embedding space, validating its theoretical grounding as a consistency measure.
• The framework achieves state-of-the-art hallucination detection performance without requiring additional models or fine-tuning.

(a) Neuron Activation