[Paper Review] A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training

Zihan Qiu, Zeyu Huang+17 more|arXiv (Cornell University)|Jan 30, 2026

Neurobiology of Language and Bilingualism0 citations

TL;DR

The paper argues that emergent outliers in attention and residual streams rescale other components via normalization, and introduces gating-based rescaling (GatedNorm) to improve training stability and quantization robustness while mitigating residual sinks.

ABSTRACT

We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations ( extit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon extit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization).

Motivation & Objective

• Investigate the functional role of attention sinks and residual sinks in large language models.
• Demonstrate that outliers interact with normalization to rescale non-outlier components.
• Show that removing normalization or naively clipping outliers harms training stability and performance.
• Propose and evaluate mitigation strategies that preserve or replace outlier-driven rescaling to improve training and quantization.

Proposed method

• Analyze outlier patterns in attention logits and residual activations across multiple models and token counts.
• Formalize the concept of outlier-driven rescaling as the interaction between outliers and normalization (softmax attention and RMSNorm).
• Conduct ablations by removing normalization, clipping outliers, or altering activation functions to assess stability and performance.
• Introduce PreAffine RMSNorm to absorb outliers into learnable parameters before normalization.
• Propose GatedNorm, an explicit gating mechanism after normalization, to enable rescaling while reducing outliers and improving quantization robustness.

Experimental results

Research questions

• RQ1 Do outliers in attention and residual streams function primarily as rescale factors rather than direct contributors to outputs?
• RQ2 Is outlier-driven rescaling essential for stable training, and can it be preserved or replaced without sacrificing performance?
• RQ3 Can explicit rescaling mechanisms (e.g., gating) mitigate residual sinks and improve robustness to quantization and architectural choices?
• RQ4 How do different normalization and attention variants (softmax, linear, gating) influence the formation of sinks and training stability?
• RQ5 Can outliers be absorbed into parameters without loss of function, and what are the trade-offs for model capacity and deployment?

Key findings

• Outliers interact with normalization to perform rescaling, and removing normalization degrades stability and performance.
• Outliers function mainly as rescale factors; their final contribution to outputs is smaller than non-outliers.
• Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, improving training and quantization robustness.
• GatedNorm reduces residual sinks and maintains or improves performance, with better quantization resilience under FP4 settings.
• Providing explicit rescaling via gating reduces reliance on outliers, making models less sensitive to architectural choices and enabling better robustness across activations and architectures.