[Paper Review] Limitations of the NTK for Understanding Generalization in Deep Learning

Nikhil Vyas, Yamini Bansal+1 more|arXiv (Cornell University)|Jun 20, 2022

Neural Networks and Applications4 citations

TL;DR

This paper demonstrates that the Neural Tangent Kernel (NTK) fails to capture key aspects of deep learning generalization, particularly data scaling laws. Despite being a theoretically grounded proxy for infinite-width networks, empirical and infinite NTKs exhibit worse data scaling exponents than finite-width neural networks across real-world datasets like CIFAR-10 and SVHN, even after partial pretraining. The study reveals that NTKs continue improving throughout training, contradicting claims of early stabilization, and thus cannot fully explain the sample efficiency of real neural networks.

ABSTRACT

The ``Neural Tangent Kernel'' (NTK) (Jacot et al 2018), and its empirical variants have been proposed as a proxy to capture certain behaviors of real neural networks. In this work, we study NTKs through the lens of scaling laws, and demonstrate that they fall short of explaining important aspects of neural network generalization. In particular, we demonstrate realistic settings where finite-width neural networks have significantly better data scaling exponents as compared to their corresponding empirical and infinite NTKs at initialization. This reveals a more fundamental difference between the real networks and NTKs, beyond just a few percentage points of test accuracy. Further, we show that even if the empirical NTK is allowed to be pre-trained on a constant number of samples, the kernel scaling does not catch up to the neural network scaling. Finally, we show that the empirical NTK continues to evolve throughout most of the training, in contrast with prior work which suggests that it stabilizes after a few epochs of training. Altogether, our work establishes concrete limitations of the NTK approach in understanding generalization of real networks on natural datasets.

Motivation & Objective

• To investigate whether the Neural Tangent Kernel (NTK) accurately captures the generalization behavior of real, finite-width neural networks.
• To evaluate the extent to which NTKs—both infinite and empirical—can explain the data and time scaling laws observed in real neural network training.
• To challenge the assumption that NTKs stabilize early in training and thus serve as a reliable proxy for representation learning.
• To identify fundamental limitations of the NTK framework in modeling the sample efficiency and generalization performance of modern deep learning models.

Proposed method

• The authors compare the data scaling behavior of finite-width neural networks, their corresponding infinite NTKs, and empirical NTKs (extracted at initialization and after various training durations) on real datasets including CIFAR-10 and SVHN.
• They use log-log scaling plots to analyze the performance of models as a function of training data size and training time, measuring scaling exponents to quantify sample efficiency.
• Empirical NTKs are extracted at multiple training steps (e.g., 10 epochs, 32 epochs) and used to re-train on fixed test sets to assess their evolving representational capacity.
• The study employs consistent hyperparameters (learning rate, batch size, optimizer) across neural networks, infinite NTKs, and empirical NTKs to ensure fair comparison under NTK parameterization.
• They perform ablation studies across learning rates, batch sizes, and optimizers to confirm robustness of findings.
• Time-scaling analysis tracks the test error of the empirical NTK extracted at different training times, revealing continuous improvement rather than early saturation.

Experimental results

Research questions

• RQ1 Do empirical and infinite NTKs exhibit the same data scaling exponents as finite-width neural networks on real datasets?
• RQ2 Can pretraining the empirical NTK on a constant number of samples close the generalization gap with real neural networks?
• RQ3 Does the empirical NTK stabilize early in training, or does it continue to improve throughout the training process?
• RQ4 How does the performance of the empirical NTK evolve over time, and does it capture the full scaling behavior of the original neural network?

Key findings

• Finite-width neural networks consistently achieve better data scaling exponents than both empirical and infinite NTKs at initialization, even under identical hyperparameters and on real datasets like CIFAR-10 and SVHN.
• As network width increases, test performance degrades and approaches that of the infinite NTK, contradicting the common belief that over-parameterization always improves generalization.
• Even after pretraining the empirical NTK on a fixed number of samples, its data scaling exponent remains worse than that of the corresponding neural network, indicating persistent limitations in representational capacity.
• The empirical NTK continues to improve uniformly throughout most of the training process, with no observed phase transition to slower learning, challenging prior claims of early stabilization.
• The extracted kernel $K^t_{ ext{fit}}$ at any fixed training time $t$ fails to match the data scaling of the original neural network, indicating that the full training trajectory is necessary to capture the correct scaling law.
• On log-log plots, the test error of the empirical NTK extracted during training shows a constant slope of improvement, with no sign of convergence or slowdown, even after 32 epochs.