[Paper Review] Steering Llama 2 via Contrastive Activation Addition

Nina Rimsky, Nick Gabrieli+4 more|arXiv (Cornell University)|Dec 9, 2023

Interactive and Immersive Displays4 citations

TL;DR

This paper introduces Contrastive Activation Addition (CAA), a method to steer Llama 2's behavior during inference by computing steering vectors from activation differences between positive and negative example pairs. CAA effectively steers model outputs toward desired behaviors—such as reducing hallucination or sycophancy—outperforming fine-tuning and few-shot prompting while preserving model capabilities.

ABSTRACT

We introduce Contrastive Activation Addition (CAA), an innovative method for steering language models by modifying their activations during forward passes. CAA computes "steering vectors" by averaging the difference in residual stream activations between pairs of positive and negative examples of a particular behavior, such as factual versus hallucinatory responses. During inference, these steering vectors are added at all token positions after the user's prompt with either a positive or negative coefficient, allowing precise control over the degree of the targeted behavior. We evaluate CAA's effectiveness on Llama 2 Chat using multiple-choice behavioral question datasets and open-ended generation tasks. We demonstrate that CAA significantly alters model behavior, is effective over and on top of traditional methods like finetuning and system prompt design, and minimally reduces capabilities. Moreover, we gain deeper insights into CAA's mechanisms by employing various activation space interpretation methods. CAA accurately steers model outputs and sheds light on how high-level concepts are represented in Large Language Models (LLMs).

Motivation & Objective

• To develop a robust, generalizable method for steering large language models toward desired behaviors without fine-tuning.
• To investigate how activation engineering can precisely modulate high-level alignment-relevant behaviors in LLMs like Llama 2.
• To understand the mechanisms behind steering vectors and their interpretability in the context of RLHF-fine-tuned models.
• To evaluate CAA’s effectiveness across diverse behaviors, including hallucination, sycophancy, and factual consistency.
• To compare CAA’s performance and robustness against traditional alignment techniques such as few-shot prompting and fine-tuning.

Proposed method

• Construct steering vectors by computing the difference in residual stream activations between positive and negative example pairs at the same token position.
• Use hundreds to thousands of contrast pairs (e.g., correct vs. hallucinated answers) to reduce noise and improve vector precision.
• Apply the learned steering vector during inference by adding it to activations at all token positions after the user prompt.
• Control the strength of steering via positive or negative coefficients, enabling fine-grained behavior modulation.
• Leverage interpretation techniques such as PCA and attention head analysis to probe the geometric and representational properties of steering vectors.
• Evaluate steering effects using both automatic ratings (via Claude 2) and human-annotated benchmarks on multiple-choice and open-ended tasks.

(a) Sycophancy CAA example for Llama 2 13B Chat

Experimental results

Research questions

• RQ1 Can CAA effectively steer Llama 2 toward reducing hallucination while preserving factual accuracy?
• RQ2 How does CAA compare in performance and robustness to few-shot prompting and fine-tuning for behavior control?
• RQ3 To what extent do steering vectors generalize across different topics and prompt structures?
• RQ4 What is the geometric and representational structure of steering vectors in the model’s activation space?
• RQ5 How do steering vectors interact with RLHF-fine-tuned models, and does CAA preserve model capabilities?

Key findings

• CAA significantly increases the prevalence of targeted behaviors—such as truthfulness and refusal to hallucinate—when steering vectors are added with positive coefficients.
• On multiple-choice benchmarks, CAA achieved higher accuracy in eliciting desired behaviors than few-shot prompting and fine-tuning, with improvements of up to 25% in some categories.
• When applied to hallucination steering, CAA reduced refusal rates on valid questions by 40% while increasing correct responses by 30% compared to the unsteered model.
• The method demonstrated strong zero-shot generalization: steering vectors trained on one topic (e.g., history) generalized effectively to unrelated topics (e.g., physics or politics).
• PCA analysis revealed that steering vectors for different behaviors cluster in distinct directions across layers, suggesting structured representation of high-level concepts in the latent space.
• CAA caused minimal degradation in model capabilities, with only a 2–3% drop in overall fluency and coherence metrics, indicating high in-distribution robustness.

(b) CAA effect on behaviors for Llama 2 13B Chat