[Paper Review] The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes

Alexander Atanasov, Blake Bordelon+2 more|arXiv (Cornell University)|Dec 23, 2022

Model Reduction and Neural Networks4 citations

TL;DR

This paper identifies the onset of variance-limited generalization in wide neural networks, showing that finite-width effects become significant at training set sizes $ P^* \sim \sqrt{N} $, where generalization degrades below infinite-width performance. The authors link this to fluctuations in the final neural tangent kernel (NTK), demonstrating that feature learning and ensemble averaging suppress variance and improve generalization beyond the critical $ P^* $.

ABSTRACT

For small training set sizes $P$, the generalization error of wide neural networks is well-approximated by the error of an infinite width neural network (NN), either in the kernel or mean-field/feature-learning regime. However, after a critical sample size $P^*$, we empirically find the finite-width network generalization becomes worse than that of the infinite width network. In this work, we empirically study the transition from infinite-width behavior to this variance limited regime as a function of sample size $P$ and network width $N$. We find that finite-size effects can become relevant for very small dataset sizes on the order of $P^* \sim \sqrt{N}$ for polynomial regression with ReLU networks. We discuss the source of these effects using an argument based on the variance of the NN's final neural tangent kernel (NTK). This transition can be pushed to larger $P$ by enhancing feature learning or by ensemble averaging the networks. We find that the learning curve for regression with the final NTK is an accurate approximation of the NN learning curve. Using this, we provide a toy model which also exhibits $P^* \sim \sqrt{N}$ scaling and has $P$-dependent benefits from feature learning.

Motivation & Objective

• To identify the critical training set size $ P^* $ at which finite-width neural networks begin to generalize worse than infinite-width networks.
• To understand the role of neural tangent kernel (NTK) variance in driving finite-width generalization errors.
• To investigate how feature learning and ensemble averaging mitigate variance-limited behavior.
• To validate the correspondence between neural network learning curves and kernel regression using the final empirical NTK (eNTK f ).
• To construct a toy random feature model that reproduces key scaling behaviors observed in real networks.

Proposed method

• Empirically study ReLU networks on polynomial regression across varying widths $ N $, training set sizes $ P $, and output scaling $ \alpha $ to tune between lazy and rich regimes.
• Use the final empirical NTK (eNTK f ) to approximate the neural network’s generalization error, showing strong correspondence with actual learning curves.
• Analyze variance of the final NTK across random initializations to quantify finite-width fluctuations.
• Apply ensemble averaging via model averaging and data augmentation to reduce variance-induced errors.
• Introduce a noisy random feature model to analytically reproduce $ P^* \sim \sqrt{N} $ scaling and alignment-based improvements from feature learning.
• Compare generalization error and variance across different network depths, input dimensions, and centering schemes to isolate architectural effects.

Experimental results

Research questions

• RQ1 At what training set size $ P $ do finite-width neural networks begin to generalize worse than infinite-width networks?
• RQ2 How does the variance of the final neural tangent kernel (NTK) drive finite-width generalization errors?
• RQ3 To what extent can feature learning and ensemble averaging suppress variance-limited behavior in finite-width networks?
• RQ4 How well does kernel regression using the final empirical NTK (eNTK f ) approximate the true neural network learning curve?
• RQ5 Can a simple random feature model reproduce the observed $ P^* \sim \sqrt{N} $ scaling and feature learning benefits?

Key findings

• The critical sample size $ P^* $ at which finite-width networks underperform infinite-width networks scales as $ P^* \sim \sqrt{N} $ for ReLU networks in polynomial regression.
• Generalization error in finite-width networks is dominated by variance over random initializations, particularly in the lazy regime, defining the variance-limited regime.
• Ensemble averaging significantly reduces variance and improves generalization, especially in the variance-limited regime.
• Feature learning—induced by small output scaling $ \alpha $ or higher-degree polynomial tasks—improves generalization both before and after $ P^* $, primarily by enhancing alignment between the final NTK and the target function.
• The learning curve of the neural network is well-approximated by kernel regression using the final empirical NTK (eNTK f ), validating its use as a proxy for network behavior.
• A toy random feature model reproduces the $ P^* \sim \sqrt{N} $ scaling and the benefits of feature learning through alignment, confirming the mechanism behind the observed phenomena.