[Paper Review] Towards Best Practices of Activation Patching in Language Models: Metrics and Methods

Fred Zhang, Neel Nanda|arXiv (Cornell University)|Sep 27, 2023

Topic Modeling4 citations

TL;DR

This paper systematically evaluates methodological choices in activation patching for mechanistic interpretability of language models, identifying critical impacts of corruption methods (Gaussian noising vs. symmetric token replacement), evaluation metrics (logit difference vs. probability), and sliding window patching. It finds that Gaussian noising disrupts model behavior and leads to inconsistent results, while logit difference better captures both positive and negative model components, recommending STR and logit difference as best practices for reliable interpretability.

ABSTRACT

Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where localization -- identifying the important model components -- is a key step. Activation patching, also known as causal tracing or interchange intervention, is a standard technique for this task (Vig et al., 2020), but the literature contains many variants with little consensus on the choice of hyperparameters or methodology. In this work, we systematically examine the impact of methodological details in activation patching, including evaluation metrics and corruption methods. In several settings of localization and circuit discovery in language models, we find that varying these hyperparameters could lead to disparate interpretability results. Backed by empirical observations, we give conceptual arguments for why certain metrics or methods may be preferred. Finally, we provide recommendations for the best practices of activation patching going forwards.

Motivation & Objective

• To investigate how methodological choices in activation patching affect interpretability results in language models.
• To evaluate the impact of different prompt corruption methods—Gaussian noising and symmetric token replacement—on localization and circuit discovery.
• To compare the effectiveness of logit difference and probability as evaluation metrics for patching effects.
• To analyze the differences between sliding window patching and single-layer patching in identifying model components.
• To provide evidence-based recommendations for best practices in activation patching.

Proposed method

• Systematically vary three key hyperparameters: prompt corruption method (Gaussian noising vs. symmetric token replacement), evaluation metric (logit difference vs. probability), and patching strategy (sliding window vs. single-layer).
• Perform activation patching across multiple tasks: factual recall, indirect object identification (IOI), greater-than comparison, Python docstring completion, and arithmetic.
• Use clean, corrupted, and patched forward passes to measure causal effects: compare model output on corrupted input vs. input with specific activations replaced from the clean run.
• Apply sliding window patching to restore activations across multiple MLP layers simultaneously, comparing results to individual layer patching.
• Evaluate results using statistical thresholds (e.g., two standard deviations from mean) to detect significant positive or negative components.
• Conduct ablation studies across GPT-J and GPT-2 small models to validate findings across architectures and tasks.

(a) Activation patching intervenes on latent states

Experimental results

Research questions

• RQ1 How do different prompt corruption methods—Gaussian noising and symmetric token replacement—affect the reliability of activation patching in localization tasks?
• RQ2 Does the choice of evaluation metric (logit difference vs. probability) lead to divergent interpretations of model component importance?
• RQ3 How does sliding window patching compare to single-layer patching in identifying functional circuits in language models?
• RQ4 Why does Gaussian noising lead to inconsistent interpretability results, and is it fundamentally problematic for causal tracing?
• RQ5 Can logit difference detect negative model components that hurt performance, and does probability fail to do so?

Key findings

• Gaussian noising leads to inconsistent localization and circuit discovery outcomes because it pushes the model out of distribution, disrupting internal mechanisms.
• Symmetric token replacement (STR) preserves in-distribution behavior and produces more reliable and consistent interpretability results than Gaussian noising.
• Logit difference detects both positive and negative model components, including those that harm performance, whereas probability can overlook negative components.
• Sliding window patching produces more pronounced and coherent localization signals than patching individual layers and summing their effects.
• STR corruption results in more stable and interpretable activation patching outcomes across multiple tasks, including factual recall and IOI circuit discovery.
• The use of logit difference as a metric leads to more precise detection of critical components, especially in identifying heads that negatively impact performance.

(b) Patching attention heads