[Paper Review] Anatomical Heterogeneity in Transformer Language Models

Tomasz Wietrzykowski|arXiv (Cornell University)|Mar 19, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper reveals strong layer-wise heterogeneity in a 30-layer SmolLM2-135M transformer, identifying a critical core, anti-layers, and a proven Growth Transformer Training paradigm that exploits per-layer importance to reduce training cost and improve performance.

ABSTRACT

Current transformer language models are trained with uniform computational budgets across all layers, implicitly assuming layer homogeneity. We challenge this assumption through empirical analysis of SmolLM2-135M, a 30-layer, 135M-parameter causal language model, using five diagnostic metrics: weight predictability (R2), ablation degradation, recovery speed, weight manipulation robustness, and structural analysis. We find profound anatomical heterogeneity: (1) Layer weights follow strong mathematical regularity (R2 = 0.91) with a universal oscillatory delta pattern (correlation ~= -0.50), yet predicted weights cause catastrophic failure due to nonlinear error accumulation. (2) Layer importance spans a 10^7 range, from a critical core (L8-11, up to +63,419% PPL degradation) to anti-layers (L14, L17) whose removal improves performance. (3) Recovery speed correlates with layer importance, indicating differential training requirements. (4) Only weight scaling (alpha = 0.9) preserves model quality among five tested manipulation strategies. (5) Growth Transformer Training, allocating budget by layer importance, achieves ~54% cost reduction. A proof-of-concept experiment confirms this: 4.7x lower validation loss than uniform training at identical parameter count, while being 13% faster.

Motivation & Objective

• Assess whether transformer layers are functionally interchangeable by measuring layer-wise impact on perplexity after ablation.
• Quantify predictability of weights across layers and assess the weight perturbation consequences.
• Characterize recovery dynamics after perturbations to infer per-layer training budget requirements.
• Identify special layer phenomena (anti-layers) and evaluate effective weight manipulation strategies.
• Propose and validate Growth Transformer Training as a practical training paradigm based on empirical layer importance.

Proposed method

• Construct a complete layer importance map for all 30 layers using five metrics including ablation degradation and recovery speed.
• Evaluate weight predictability with Ridge regression on compressed weight samples and analyze the R^2 vs. functional degradation.
• Analyze weight structure via delta correlations across adjacent layers and PCA across layers.
• Test five weight manipulation strategies on redundant layers to assess preservation of generation quality.
• Conduct a proof-of-concept Growth Transformer Training with six developmental phases to compare against a uniform baseline.

Experimental results

Research questions

• RQ1 Is there a functional hierarchy among transformer layers that enables differential training budgets?
• RQ2 Can later-layer weights be predicted from earlier weights without enabling functional interchangeability?
• RQ3 Do layers differ in recovery speed after perturbation, indicating differential training needs?
• RQ4 Are there anti-layers whose perturbation can improve model performance?
• RQ5 Can Growth Transformer Training outperform uniform training in terms of loss, speed, and efficiency?

Key findings

• Layer importance spans a wide range, with a critical core (L8–L11) and anti-layers (L14, L17) that degrade or improve performance upon perturbation.
• R^2 of weight predictability is high (e.g., 0.909 for mlp.gate_proj) yet predicted weights catastrophically degrade perplexity when substituted, due to nonlinear error accumulation.
• Inter-layer weight changes follow a universal oscillatory pattern with delta correlation ≈ -0.50 across all components, suggesting a compensatory mechanism in residual connections.
• Among five weight manipulation strategies, only weight scaling by 0.9 preserves generation quality for redundant layers; others explode perplexity.
• Growth Transformer Training, a six-phase developmental protocol, achieves 4.7x lower validation loss than uniform training at the same steps and 13% faster in the proof-of-concept experiment, and 2.1x lower loss at half budget.