[Paper Review] Steering Language Models With Activation Engineering

Alexander Turner, Lisa Thiergart+5 more|arXiv (Cornell University)|Aug 20, 2023

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

This paper introduces Activation Addition (ActAdd), a method to steer large language models at inference time by computing steering vectors from activation differences between prompt pairs, without optimization or labeled data. It enables natural language-controlled behavior modification—such as sentiment, topic, or style—while preserving model performance and scaling efficiently with model size.

ABSTRACT

Prompt engineering and finetuning aim to maximize language model performance on a given metric (like toxicity reduction). However, these methods do not fully elicit a model's capabilities. To reduce this gap, we introduce activation engineering: the inference-time modification of activations in order to control (or steer) model outputs. Specifically, we introduce the Activation Addition (ActAdd) technique, which contrasts the intermediate activations on prompt pairs (such as "Love" versus "Hate") to compute a steering vector (Subramani et al. 2022). By tactically adding in e.g. the "Love" - "Hate" steering vector during the forward pass, we achieve SOTA on negative-to-positive sentiment shift and detoxification using models including LLaMA-3 and OPT. ActAdd yields inference-time control over high-level output properties (like topic and sentiment) while preserving performance on off-target tasks. ActAdd is lightweight: it does not require any machine optimization and works with a single pair of data points, which enables rapid iteration over steering. ActAdd demonstrates the power of activation engineering.

Motivation & Objective

• Address the challenge of reliably controlling large language models (LLMs) without computationally expensive fine-tuning or reinforcement learning.
• Enable user-friendly, natural language specification of desired model behaviors during inference.
• Develop a lightweight, scalable method for steering LLMs that avoids backward passes and labeled data.
• Preserve overall model performance while enabling targeted control over high-level output properties such as sentiment, topic, and style.

Proposed method

• Compute a steering vector as the difference between activations produced by two contrasting prompts (e.g., 'I love you' vs. 'I hate you').
• Apply the steering vector to the residual stream of the model during inference by adding it to activations at a specific layer.
• Scale the steering vector by a learnable coefficient to control the strength of the behavioral shift.
• Use only forward passes—no backward passes, fine-tuning, or labeled data are required to generate the steering vector.
• Apply the method at inference time on frozen LLMs, enabling real-time control without retraining.
• Select the layer and coefficient via ablation or heuristic search to maximize desired behavioral changes.

Figure 1: Schematic of the Activation Addition ( ActAdd ) method. $=$ natural language text; $\bullet$ = vectors of activations just before a specified layer. In this example, the output is heavily biased towards discussing weddings, regardless of the topic of the user prompt. (See Algorithm 1 for o

Experimental results

Research questions

• RQ1 Can activation differences between natural language prompt pairs reliably generate effective steering vectors for inference-time control?
• RQ2 Does ActAdd preserve the model's overall performance on out-of-distribution or non-targeted tasks?
• RQ3 How does the method scale with model size and what is its computational overhead compared to fine-tuning or RLHF?
• RQ4 Can ActAdd achieve monotonic or non-monotonic control over specific attributes like sentiment or topic, and what explains such behavior?
• RQ5 Is the method robust across different models, such as GPT-2, Llama-13B, and GPT-J-6B, and does it generalize across diverse prompts?

Key findings

• ActAdd successfully steers GPT-2 on OpenWebText and ConceptNet to alter sentiment, topic, and style with minimal computational cost and no fine-tuning.
• The method preserves model performance on non-targeted tasks, demonstrating robustness and generalization.
• On GPT-2, ActAdd achieves significant control over output properties—e.g., shifting completions from 'hate' to 'love'—using only forward passes and no labeled data.
• Non-monotonic behavior was observed when modifying fractions of residual stream dimensions, suggesting complex feature representations and potential axis-alignment in activation space.
• The approach scales naturally with model size and is significantly easier to implement than other activation engineering methods requiring gradient-based optimization.
• Replication on Llama-13B shows similar success, though some prompts (e.g., Eiffel Tower fact) showed failed replication, indicating sensitivity to context and feature alignment.

Figure 2: Performance of ActAdd on a target topic as the topic becomes more relevant. The perplexity ratio (lower better) compares the relative predictive performance of ActAdd and an unmodified model; we see that adding a wedding - ‘ ’ steering vector improves performance on wedding-related text wh