[Paper Review] Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden States

Hanyu Duan, Yi Yang+1 more|arXiv (Cornell University)|Feb 15, 2024

Insurance and Financial Risk Management4 citations

TL;DR

This paper investigates whether large language models (LLMs) are aware of hallucination by analyzing their hidden states during response generation. Using a controlled experimental framework comparing correct and hallucinated answers on LLaMA models, the authors find that LLMs exhibit distinct hidden state dynamics for truthful vs. fabricated responses, enabling the derivation of a hallucination-awareness metric and a novel mitigation strategy via activation engineering using learned correct-response directions.

ABSTRACT

Large Language Models (LLMs) can make up answers that are not real, and this is known as hallucination. This research aims to see if, how, and to what extent LLMs are aware of hallucination. More specifically, we check whether and how an LLM reacts differently in its hidden states when it answers a question right versus when it hallucinates. To do this, we introduce an experimental framework which allows examining LLM's hidden states in different hallucination situations. Building upon this framework, we conduct a series of experiments with language models in the LLaMA family (Touvron et al., 2023). Our empirical findings suggest that LLMs react differently when processing a genuine response versus a fabricated one. We then apply various model interpretation techniques to help understand and explain the findings better. Moreover, informed by the empirical observations, we show great potential of using the guidance derived from LLM's hidden representation space to mitigate hallucination. We believe this work provides insights into how LLMs produce hallucinated answers and how to make them occur less often.

Motivation & Objective

• To investigate whether LLMs possess internal awareness of hallucination by analyzing their hidden representations.
• To compare hidden state transitions when processing correct versus hallucinated answers under identical question conditions.
• To develop a quantifiable measure of LLM hallucination awareness based on hidden state differences.
• To explore whether guidance derived from hidden state representations can reduce hallucination in LLM outputs.
• To evaluate the effectiveness of activation engineering using learned correct-response directions to steer LLMs toward truthful generations.

Proposed method

• Design a two-input experimental framework: one with a correct answer and one with a hallucinated answer, both sharing the same question.
• Extract three key hidden states: s₁ (question-only), s₂ (question + hallucinated answer), and s₃ (question + correct answer) from the final transformer layer.
• Use PCA to identify dominant directions in hidden state space: d_corr (correct transition) and d_halluc (hallucinated transition).
• Apply activation engineering by adding a scaled d_corr vector (α=100) to the final hidden state at each decoding step to guide responses toward truthfulness.
• Compare model outputs with and without the d_corr offset on TruthfulQA and HaluEval datasets to evaluate mitigation effectiveness.
• Use interpretability techniques (e.g., PCA, attention analysis) to explain hidden state differences and validate the awareness metric.

Figure 1: Experimental framework, including two independent inputs (i.e., hallucinated input and correct input) and three critical hidden states ( $\bm{s_{1}}$ , $\bm{s_{2}}$ , and $\bm{s_{3}}$ ). <Question> and <Answer> are templates that are adaptable and can be customized to suit various tasks.

Experimental results

Research questions

• RQ1 Do LLMs exhibit different hidden state dynamics when processing a correct answer versus a hallucinated one?
• RQ2 To what extent can hidden state representations reflect an LLM’s internal awareness of hallucination?
• RQ3 Can the direction of hidden state transition toward a correct response be identified and used to guide model behavior?
• RQ4 Does injecting the correct-response transition direction into hidden states reduce hallucination in LLM generations?
• RQ5 How does the level of hallucination awareness correlate with model uncertainty and response reliability?

Key findings

• LLMs show significantly different hidden state transitions when processing correct vs. hallucinated answers, indicating internal awareness of hallucination.
• The final hidden state is more sensitive to correct answers than to hallucinated ones, suggesting stronger internal signal alignment with truthfulness.
• A hallucination-awareness metric derived from hidden state differences correlates with model uncertainty and response reliability.
• Strategically injecting the correct-response transition direction (d_corr) into hidden states reduces hallucination, with 100% of tested examples showing improved alignment with ground truth.
• The activation engineering approach successfully reversed incorrect responses to correct ones in 33% of cases (e.g., example #1), and improved conciseness and completeness in others.
• The effect size difference (e_corr - e_halluc) between correct and hallucinated transitions is most pronounced in middle to upper transformer layers, with 95% confidence intervals showing consistent trends.

Figure 2: Awareness score distributions. We mark the average score for each model in red.