[Paper Review] Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers

Zeyuan Allen-Zhu, Yuanzhi Li+1 more|arXiv (Cornell University)|Nov 12, 2018

Machine Learning and Algorithms49 references172 citations

TL;DR

The paper proves that overparameterized neural networks can efficiently learn concept classes beyond two layers using SGD, with results for both two- and three-layer networks and a learning approach that goes beyond NTK. It introduces a quadratic (second-order) approximation framework and shows polynomial-time learnability with sample complexity almost independent of network size.

ABSTRACT

The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized? In this work, we prove that overparameterized neural networks can learn some notable concept classes, including two and three-layer networks with fewer parameters and smooth activations. Moreover, the learning can be simply done by SGD (stochastic gradient descent) or its variants in polynomial time using polynomially many samples. The sample complexity can also be almost independent of the number of parameters in the network. On the technique side, our analysis goes beyond the so-called NTK (neural tangent kernel) linearization of neural networks in prior works. We establish a new notion of quadratic approximation of the neural network (that can be viewed as a second-order variant of NTK), and connect it to the SGD theory of escaping saddle points.

Motivation & Objective

• Motivate the fundamental questions of what functions neural networks can provably learn and why overparameterization aids generalization.
• Show that smooth two- and three-layer networks can be learned efficiently by SGD with polynomial time and sample complexity.
• Develop and leverage a quadratic-approximation (second-order NTK-like) framework to analyze SGD dynamics beyond standard NTK.
• Demonstrate that overparameterization allows learning target functions with potentially complex activations and smaller population risk.

Proposed method

• Formulate a target function class that includes two- and three-layer networks with smooth activations.
• Analyze SGD dynamics on overparameterized networks initialized with Gaussian weights and trained to minimize a convex/1-Lipschitz loss.
• Introduce a quadratic-approximation perspective as a second-order variant of NTK to study SGD escaping saddle points.
• In the three-layer setting, employ a weight-decay regularization and a two-stage SGD with Gaussian perturbations to ensure favorable landscape properties.
• Prove that, given appropriate initialization and sample size, SGD achieves population risk within ε of the best target function in the class (OPT+ε).
• Provide polynomial-time and polynomial-sample-size bounds that are almost independent of the number of parameters.

Experimental results

Research questions

• RQ1 What classes of functions can neural networks with more than two layers provably learn under overparameterization?
• RQ2 Can SGD efficiently learn these classes in polynomial time with sample sizes that do not grow with parameter counts?
• RQ3 How does moving beyond NTK (nonlinear interactions across layers) influence learnability and generalization?
• RQ4 What roles do overparameterization and regularization (weight decay) play in enabling generalization for deeper networks?
• RQ5 How does the proposed quadratic approximation relate to SGD dynamics and saddle-point escape in multi-layer networks?

Key findings

• Overparameterized networks can efficiently learn concept classes containing two- and three-layer networks with non-trivial activations.
• Learning is achievable by SGD or its variants in polynomial time with polynomially many samples.
• Sample complexity is almost independent of the number of parameters (m) and depends on activation complexity and target network size.
• A new quadratic-approximation framework (second-order NTK) connects SGD analysis to escaping saddle points beyond kernel-style linearization.
• For two-layer networks, SGD can achieve population risk OPT+ε with a polynomial-time bound and sample complexity that scales with activation complexity and target size.
• For three-layer networks, the results extend beyond NTK, showing learnability of more expressive target functions under overparameterization and regularization.