[Paper Review] The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers

Zonglin Li, Chong You+9 more|arXiv (Cornell University)|Oct 12, 2022

Neural Networks and Applications10 citations

TL;DR

This paper identifies and investigates the 'lazy neuron' phenomenon in trained Transformer models, where intermediate MLP activations become highly sparse—only 3–6% of entries are nonzero—despite dense computation. The sparsity emerges naturally during training across diverse architectures and datasets, and enforcing even stricter sparsity via Top-k pruning improves model robustness, calibration, and efficiency without performance loss.

ABSTRACT

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by sparse we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

Motivation & Objective

• Understand why intermediate activations in trained Transformers become highly sparse despite dense computation.
• Investigate whether this sparsity is a result of dataset-specific properties or a general phenomenon across architectures and data types.
• Explore the implications of sparsity for model efficiency, robustness, and calibration.
• Determine whether enforced sparsity (via Top-k pruning) can improve model generalization and reliability.
• Explain the emergence of sparsity through training dynamics rather than explicit regularization.

Proposed method

• Measure activation sparsity as the percentage of nonzero entries in ReLU-activated MLP outputs across multiple Transformer architectures and tasks.
• Train models on natural, random, and infinite data to isolate causes of sparsity from data distribution effects.
• Use two-layer MLPs with varying widths and optimizers to study sparsity in simpler models and observe uni-modal sparsity patterns.
• Apply Top-k pruning to enforce extreme sparsity and evaluate downstream robustness and calibration.
• Analyze gradient dynamics at initialization to link sparsity emergence to training dynamics.
• Compare sparsity across different model depths, widths, and training regimes (train/eval data).

Experimental results

Research questions

• RQ1 To what extent is activation sparsity in Transformers a general phenomenon across different architectures, tasks, and data distributions?
• RQ2 Does sparsity emerge due to dataset characteristics or is it driven by training dynamics?
• RQ3 What is the relationship between model width, parameterization regime (under/over/interpolation), and activation sparsity?
• RQ4 Can enforcing extreme sparsity via Top-k pruning improve model robustness and confidence calibration?
• RQ5 Why does sparsity emerge in Transformers without explicit regularization or sparsity-inducing objectives?

Key findings

• Sparsity in Transformer MLP activations is highly prevalent: T5-Base exhibits only 2.7% nonzero entries on average, and ViT-B16 shows 6.3%.
• Sparsity increases with model depth and width, indicating a stronger emergence in larger models.
• Sparsity persists across both natural and random data, including infinite data settings, indicating it is not data-dependent.
• Sparsity emerges even in simple two-layer MLPs, suggesting it is a property of training dynamics rather than architectural design.
• Top-k pruning with small k (e.g., k=1 or 2) improves robustness to input corruptions and reduces sensitivity to noisy labels.
• Sparsity correlates with better confidence calibration and enables significant FLOP reduction without accuracy drop.