[Paper Review] Discovering Latent Knowledge in Language Models Without Supervision

Collin Burns, Haotian Ye+2 more|arXiv (Cornell University)|Dec 7, 2022

Topic Modeling45 citations

TL;DR

The paper presents Contrast-Consistent Search (CCS), an unsupervised method to extract latent truth representations from language-model activations to answer yes-no questions, outperforming zero-shot baselines on average and reducing prompt sensitivity.

ABSTRACT

Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4\% on average. We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.

Motivation & Objective

• Motivate and formalize the problem of extracting latent truth from language models without supervision.
• Develop a lightweight probe that identifies a truth-related direction in activation space.
• Demonstrate that this latent knowledge transfers across tasks and is robust to misleading prompts.
• Analyze characteristics of the learned representations and their data/sample efficiency.

Proposed method

• Construct contrast pairs by formatting each yes-no question as both a positive and negative statement.
• Extract model activations for each contrastive pair and normalize them.
• Map normalized activations to probabilities via a linear probe with a sigmoid activation.
• Train an unsupervised loss combining a consistency term (p(x+)=1-p(x−)) and a confidence term to avoid degenerate solutions.
• Infer answers by averaging p(x+) and 1−p(x−) and choosing the >0.5 decision boundary.

Experimental results

Research questions

• RQ1 Can latent truth representations in language models be discovered without supervision from activations alone?
• RQ2 Do such representations generalize across datasets and tasks beyond the training data?
• RQ3 Are these representations robust to manipulations of model outputs or misleading prompts?
• RQ4 Where in the model (which layers) do these truth representations reside and how data-efficient are they?
• RQ5 How do the discovered representations relate to the model’s own outputs and ground-truth labels?

Key findings

• CCS outperforms strong zero-shot baselines by an average of 4 percentage points across 6 models and 10 datasets.
• CCS reduces prompt sensitivity, with average accuracy more robust across different prompts.
• Misleading prompts that degrade zero-shot performance do not significantly affect CCS accuracy.
• Latent truth representations transfer across datasets and tasks, indicating a task-agnostic truth direction.
• Intermediate layers often yield better CCS performance than final outputs, suggesting latent knowledge beyond what is in the outputs.
• Finding truth representations can be data-efficient, sometimes workable with very few contrast pairs.