[Paper Review] LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

Hadas Orgad, Michael Toker+5 more|arXiv (Cornell University)|Oct 3, 2024

Blockchain Technology Applications and Security4 citations

TL;DR

This paper reveals that large language models (LLMs) encode significantly more intrinsic truthfulness information in their internal representations than previously recognized, particularly concentrated in exact answer tokens. By training probing classifiers on these representations, the authors achieve superior error detection and discover that truthfulness signals are task-specific rather than universal, enabling prediction of error types and exposing a critical disconnect between internal knowledge and external generation behavior.

ABSTRACT

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

Motivation & Objective

• To investigate the intrinsic representation of hallucinations in LLMs beyond surface-level error detection.
• To identify where in the model's internal states truthfulness information is most concentrated.
• To evaluate whether truthfulness signals generalize across different tasks and datasets.
• To explore whether internal representations can predict not just errors, but also error types.
• To examine the alignment between internal knowledge and actual model behavior, especially when correct answers are encoded but not generated.

Proposed method

• Training probing classifiers on intermediate layer representations of LLMs, focusing specifically on the tokens corresponding to the exact answer in generated outputs.
• Using multiple datasets (TriviaQA, Math, Winobias) to evaluate error detection performance across diverse tasks.
• Comparing the performance of probing classifiers against logit-based uncertainty predictors to assess generalization and robustness.
• Analyzing repeated generations to classify error patterns (e.g., consistent, occasional, competing answers) and train classifiers on internal representations to predict these types.
• Employing a taxonomy of error types based on response consistency across multiple samples to link internal representations to behavioral patterns.
• Measuring the discrepancy between internal encoding of correct answers and actual model output to reveal internal-external misalignment.

Figure 1: Example for the input and LLM output from the TriviaQA dataset, and the names of the tokens that can be probed.

Experimental results

Research questions

• RQ1 Where in the LLM’s internal representation is truthfulness information most concentrated?
• RQ2 Can probing classifiers trained on internal representations detect hallucinations more effectively than existing methods?
• RQ3 Do truthfulness signals in LLMs generalize across different tasks, or are they task-specific?
• RQ4 Can internal representations predict the type of error a model is likely to generate?
• RQ5 To what extent is there a disconnect between the model’s internal knowledge of the correct answer and its actual output?

Key findings

• Truthfulness information is most strongly encoded in the exact answer tokens, and focusing on these tokens significantly improves error detection performance.
• Probing classifiers trained on answer tokens achieve state-of-the-art detection performance, with F1 scores up to 0.79 on TriviaQA and 0.83 on Winobias.
• Error detection models fail to generalize across tasks, indicating that truthfulness encoding is not universal but rather skill-specific.
• Internal representations can predict error types (e.g., consistent incorrect, competing answers), enabling targeted mitigation strategies.
• A significant discrepancy exists between internal encoding of correct answers and actual model generation, where models may know the truth but still produce hallucinated outputs.

(a) TriviaQA