[Paper Review] How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective

Runyu Peng, Renzhi Li+4 more|arXiv (Cornell University)|Feb 4, 2026

Artificial Intelligence in Healthcare and Education0 citations

TL;DR

The paper identifies a simple two-layer P0-Sink Circuit that makes position-zero tokens acquire a stable, high-norm representation, causing attention sinks in LLMs, and tracks its emergence during pre-training.

ABSTRACT

Large Language Models (LLMs) often allocate disproportionate attention to specific tokens, a phenomenon commonly referred to as the attention sink. While such sinks are generally considered detrimental, prior studies have identified a notable exception: the model's consistent emphasis on the first token of the input sequence. This structural bias can influence a wide range of downstream applications and warrants careful consideration. Despite its prevalence, the precise mechanisms underlying the emergence and persistence of attention sinks remain poorly understood. In this work, we trace the formation of attention sinks around the first token of the input. We identify a simple mechanism, referred to as the P0 Sink Circuit, that enables the model to recognize token at position zero and induce an attention sink within two transformer blocks, without relying on any semantic information. This mechanism serves as the basis for the attention sink on position zero. Furthermore, by analyzing training traces from a 30B A3B MoE model trained from scratch, we find that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers, suggesting a possible signal for tracking pre training convergence states.

Motivation & Objective

• Investigate why the position-zero attention sink emerges in large language models.
• Determine whether the BOS token drives the P0 sink or a deeper architectural mechanism exists.
• Propose and validate a simple circuit that identifies position zero and amplifies its representation.
• Trace the formation and stabilization of the P0-Sink Circuit during pre-training and across model layers.
• Assess potential uses of sink dynamics as indicators of pretraining convergence and training stage.

Proposed method

• Introduce the P0-Sink Circuit as a two-layer transformer mechanism leveraging causal-attention masking asymmetry.
• Show that position-zero can be identified and amplified into a high-norm fixed representation without relying on BOS semantics.
• Use ablation studies removing BOS to test dependence on BOS semantics.
• Provide a theoretical model of attention output norm using a cone-based value-vector construction and uniform averaging across heads.
• Empirically analyze training traces from a 30B-A3B MoE model to track emergence and consolidation of the P0-Sink Circuit across layers.
• Examine how pre-layer normalization and MLP amplification stabilize the P0 representation.

Experimental results

Research questions

• RQ1 What mechanisms underlie the emergence of the position-zero attention sink in LLMs?
• RQ2 Is the BOS token essential for the P0 sink, or does a BOS-independent circuit exist?
• RQ3 How does the P0-Sink Circuit form and concentrate across layers during pre-training?
• RQ4 Can the P0-Sink be used as a diagnostic signal for training stage or convergence?

Key findings

• A simple P0-Sink Circuit exists: a two-layer mechanism that uses causal-masking asymmetry to identify position zero and amplify its hidden state into a fixed direction with high norm.
• The P0 sink persists even after removing BOS, indicating it is not solely due to BOS semantics.
• The P0-Sink Circuit emerges early in pre-training, first in deeper layers, then concentrates in the first two layers, and later transitions to broader early-layer sinks.
• MLP sublayers contribute to norm amplification and directional projection, enabling stable P0 representation across layers.
• Under relative/rotary positional encoding, the P0 sink remains robust to out-of-distribution perturbations and token repeats, supporting a BOS-independent mechanism.
• The findings suggest a potential diagnostic use: the sink stage can indicate pretraining convergence status and training progression.