[Paper Review] Explaining Neural Scaling Laws

Yasaman Bahri, Ethan Dyer+3 more|arXiv (Cornell University)|Feb 12, 2021

Neural Networks and Applications79 references31 citations

TL;DR

The paper proposes a theory that explains four neural scaling regimes (variance-limited and resolution-limited for both dataset and model size), linking exponents to data-manifold intrinsic dimension and kernel spectra, with empirical validation on random-feature and pretrained models across standard datasets.

ABSTRACT

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origins of and relationships between scaling exponents.

Motivation & Objective

• Explain why neural networks exhibit power-law scaling with dataset size and parameter count.
• Connect scaling exponents to data distribution, intrinsic data-manifold dimension, and kernel spectra.
• Provide a unified framework that includes variance-limited and resolution-limited regimes for both D and P.
• Demonstrate the theory with solvable linear/random-feature models and empirical experiments on standard architectures/datasets.

Proposed method

• Define four scaling regimes: variance-limited and resolution-limited for dataset size D and parameter count P (or width w).
• Develop theoretical arguments: variance-limited exponents arise from smooth infinite-data or infinite-width limits; resolution-limited exponents arise from models partitioning a data manifold and kernel spectra.
• Present solvable linear/random-feature teacher-student models to realize all four regimes and derive loss formulas (e.g., L(P) and L(D)).
• Relate resolution-limited exponents to intrinsic data-manifold dimension d and to kernel spectral decay (lambda_i ~ i^-(1+alpha_K)).
• Demonstrate duality between under-parameterized/over-parameterized regimes and link exponents via kernel spectra.
• Support theory with experiments on random features and pretrained models across standard datasets and architectures.

Experimental results

Research questions

• RQ1 What are the scaling regimes of neural networks with respect to dataset size D and parameter count P?
• RQ2 How do the scaling exponents depend on data distribution, intrinsic data-manifold dimension d, and kernel spectra?
• RQ3 Can a unified theory explain both variance-limited and resolution-limited scaling across different model regimes?
• RQ4 How do architectural choices and data preprocessing (noise, superclassing) affect scaling exponents?
• RQ5 Do simple solvable models (linear/random-feature) reproduce all four scaling behaviors observed in real networks?

Key findings

• Four scaling regimes are identified and empirically supported: variance-limited with universal exponents and resolution-limited with data-distribution-dependent exponents for both D and P.
• In variance-limited regimes, exponents are universal (alpha_D = alpha_W = 1) under suitable conditions.
• In resolution-limited regimes, exponents depend on the data distribution and are related to the intrinsic dimension d of the data manifold and kernel spectral decay.
• A duality links under-parameterized and over-parameterized regimes through kernel spectra and data-points projections.
• Random-feature and pretrained-model experiments reproduce the four regimes, with exponents influenced by dataset, architecture, and input distribution.
• Changing input distribution via noise or dataset can significantly alter exponents, while superclassing targets has limited effect.