[Paper Review] Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

Mert Yüksekgönül, Varun Chandrasekaran+6 more|arXiv (Cornell University)|Sep 26, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes a constraint-satisfaction lens to analyze factual errors in LLMs by modeling factual queries as Constraint Satisfaction Problems (CSPs). It introduces SAT Probe, a method that uses attention patterns to predict factual accuracy and errors early in inference, showing strong correlation between attention to constraint tokens and correctness across Llama-2 models (7B–70B) on 11 datasets with 40k+ prompts.

ABSTRACT

We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems and use this framework to investigate how the LLM interacts internally with factual constraints. We find a strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations. We curate a suite of 10 datasets containing over 40,000 prompts to study the task of predicting factual errors with the Llama-2 family across all scales (7B, 13B, 70B). We propose SAT Probe, a method probing attention patterns, that can predict factual errors and fine-grained constraint satisfaction, and allow early error identification. The approach and findings take another step towards using the mechanistic understanding of LLMs to enhance their reliability.

Motivation & Objective

• To understand the internal mechanisms behind factual errors in large language models (LLMs), particularly in Transformer-based architectures.
• To address the gap in mechanistic understanding of how LLMs fail on factual reasoning tasks, despite strong performance on correct factual recall.
• To develop a method for early detection of factual errors using model internals, reducing reliance on expensive post-hoc verification.
• To investigate the role of attention mechanisms in constraint satisfaction and factual correctness during generation.

Proposed method

• Model factual queries as Constraint Satisfaction Problems (CSPs), where each query imposes constraints (e.g., subject-relation-object triples) that the response must satisfy.
• Define constraint tokens in the input (e.g., 'director', 'born in') as key indicators of factual structure, and track their attention patterns across layers.
• Propose SAT Probe, a simple linear probe applied to attention activations targeting constraint tokens to predict constraint satisfaction and factual accuracy.
• Train and evaluate SAT Probe on a curated benchmark of 11 datasets with over 40,000 prompts spanning single- and multi-constraint queries.
• Use early-layer attention statistics to predict factual errors halfway through the forward pass, enabling cost-saving early stopping.
• Evaluate performance using accuracy, F1, and AUC metrics, comparing SAT Probe to LLM confidence scores and baseline methods.

Figure 1: Tracking attention to predict constraint satisfaction and factual errors. We view factual queries as Constraint Satisfaction Problems. That is, factual queries impose a set of constraints that the LLM’s responses must satisfy. To predict constraint satisfaction (i.e., factual correctness),

Experimental results

Research questions

• RQ1 How does attention to constraint tokens correlate with factual correctness in LLM-generated responses?
• RQ2 Can a simple probe on attention patterns predict factual errors earlier than standard LLM confidence scoring?
• RQ3 Does the model’s internal attention behavior reflect constraint satisfaction, even when the final output is factually incorrect?
• RQ4 How does the popularity of a constraint (e.g., common vs. rare entities) affect attention patterns and factual accuracy?
• RQ5 Can SAT Probe be used to identify and halt factually incorrect generations early in inference to save computational cost?

Key findings

• A strong positive correlation exists between attention to constraint tokens and factual correctness: lower attention to constraints predicts higher factual error rates.
• SAT Probe achieves comparable predictive performance to the LLM’s own confidence score in detecting factual errors, with AUC scores above 0.85 on most datasets.
• For the Llama-2 70B model, SAT Probe achieves an AUC of 0.86 on the 'Movies directed by' task and 0.83 on 'Songs performed by', indicating high reliability.
• SAT Probe can predict factual errors with high accuracy using only the first half of the forward pass, enabling early stopping and computational savings.
• On the CounterFact dataset, SAT Probe shows robust performance even for low-popularity constraints, with AUC values ranging from 0.58 to 0.71 across different relations.
• The method generalizes across model scales (7B, 13B, 70B), demonstrating consistent performance across different parameter regimes.