[Paper Review] A Note on Lazy Training in Supervised Differentiable Programming

Lénaïc Chizat, Francis Bach|arXiv (Cornell University)|Dec 5, 2018

Machine Learning and ELMComputer Science87 citations

TL;DR

This paper demonstrates that lazy training in over-parameterized neural networks effectively reduces to kernel regression due to a scaling choice that linearizes the model around initialization, rather than over-parameterization per se. The authors show this behavior emerges from weight initialization scaling, not model size, and argue it is unlikely to explain the success of deep learning in high-dimensional tasks.

ABSTRACT

In a series of recent theoretical works, it has been shown that strongly over-parameterized neural networks trained with gradient-based methods could converge linearly to zero loss, with their parameters hardly varying. In this note, our goal is to exhibit the simple structure that is behind these results. In a simplified setting, we prove that lazy training essentially solves a kernel regression. We also show that this behavior is not so much due to over-parameterization than to a choice of scaling, often implicit, that allows to linearize the model around its initialization. These theoretical results complemented with simple numerical experiments make it seem unlikely that lazy training is behind the many successes of neural networks in high dimensional tasks.

Motivation & Objective

• To identify the underlying structural reason for the linear convergence of over-parameterized neural networks in gradient-based training.
• To clarify whether over-parameterization or weight scaling is the key factor enabling lazy training behavior.
• To demonstrate that lazy training is equivalent to kernel regression in a simplified setting.
• To challenge the assumption that lazy training explains the success of deep learning in high-dimensional tasks.

Proposed method

• Analyzing a simplified neural network setup with gradient-based optimization.
• Introducing a scaling factor in weight initialization that linearizes the model around its initial values.
• Proving that the training dynamics converge to a solution equivalent to kernel ridge regression.
• Using theoretical analysis to show that the linearization is due to scaling, not over-parameterization.
• Conducting minimal numerical experiments to validate the theoretical findings.
• Comparing the behavior of the model under different scaling regimes to isolate the effect of initialization scaling.

Experimental results

Research questions

• RQ1 What structural property underlies the linear convergence of over-parameterized neural networks in gradient training?
• RQ2 Is over-parameterization or weight scaling the primary driver of lazy training behavior?
• RQ3 To what extent does lazy training reduce to kernel regression in simplified settings?
• RQ4 Can the equivalence between lazy training and kernel methods be formally established?
• RQ5 Why is lazy training unlikely to explain the success of neural networks in high-dimensional tasks?

Key findings

• Lazy training in over-parameterized networks is fundamentally due to a scaling of the initial weights that linearizes the model around initialization.
• The training dynamics in this regime are mathematically equivalent to kernel ridge regression with a neural tangent kernel.
• The convergence to zero loss occurs linearly, not because of model over-parameterization, but due to the chosen scaling of initialization.
• Numerical experiments confirm that the linearized behavior holds under the specified scaling, even with minimal over-parameterization.
• The equivalence to kernel regression suggests that the model's performance is constrained by kernel methods, limiting its capacity to generalize in high-dimensional tasks.
• The results imply that lazy training is unlikely to be the mechanism behind the empirical success of deep neural networks in complex, high-dimensional learning problems.