[Paper Review] A Convergence Theory for Deep Learning via Over-Parameterization

Zeyuan Allen-Zhu, Yuanzhi Li+1 more|arXiv (Cornell University)|Nov 9, 2018

Reinforcement Learning in Robotics55 references627 citations

TL;DR

The paper proves that over-parameterized deep neural networks trained with SGD/gradient descent from random initialization can achieve zero training error (or 100% training accuracy) in polynomial time under mild assumptions, by showing near-convexity and NTK equivalence in a large neighborhood of initialization.

ABSTRACT

Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely unsettled. In this work, we prove why stochastic gradient descent (SGD) can find $ extit{global minima}$ on the training objective of DNNs in $ extit{polynomial time}$. We only make two assumptions: the inputs are non-degenerate and the network is over-parameterized. The latter means the network width is sufficiently large: $ extit{polynomial}$ in $L$, the number of layers and in $n$, the number of samples. Our key technique is to derive that, in a sufficiently large neighborhood of the random initialization, the optimization landscape is almost-convex and semi-smooth even with ReLU activations. This implies an equivalence between over-parameterized neural networks and neural tangent kernel (NTK) in the finite (and polynomial) width setting. As concrete examples, starting from randomly initialized weights, we prove that SGD can attain 100% training accuracy in classification tasks, or minimize regression loss in linear convergence speed, with running time polynomial in $n,L$. Our theory applies to the widely-used but non-smooth ReLU activation, and to any smooth and possibly non-convex loss functions. In terms of network architectures, our theory at least applies to fully-connected neural networks, convolutional neural networks (CNN), and residual neural networks (ResNet).

Motivation & Objective

• Motivate theoretical understanding of why deep networks trained by first-order methods succeed in practice despite non-convex, non-smooth objectives.
• Show that over-parameterized deep networks can be trained to zero training error in polynomial time from random initialization.
• Extend over-parameterization theory from two-layer to multi-layer networks, including ReLU activations and various architectures.
• Establish connections between over-parameterized networks and the neural tangent kernel (NTK) in finite, polynomial width.
• Provide a framework that applies to fully-connected, CNN, and residual network architectures under mild data assumptions.

Proposed method

• Analyze training dynamics of L-layer fully-connected networks with ReLU activations under ℓ2 regression (and extendable to other losses).
• Prove that near random initialization the objective is almost convex and semi-smooth, enabling SGD/GD to converge in polynomial time.
• Show equivalence between over-parameterized networks and NTK in finite width (m = poly(L)) rather than infinite width.
• Derive gradient formulas and backpropagation structure with sign matrices D_i,ℓ to handle ReLU non-smoothness.
• Demonstrate that forward/backward propagations remain controlled (no exponential gradient explosion or vanishing) across L layers.
• Provide a stability analysis against small perturbations and discuss implications for generalization via NTK behavior.

Experimental results

Research questions

• RQ1 Can deep neural networks trained by SGD from random initialization achieve zero training error under mild over-parameterization and non-degenerate data?
• RQ2 How large must the hidden width be (as a polynomial in n, L, and data separation δ) to guarantee polynomial-time convergence?
• RQ3 Does the training landscape exhibit near-convexity and semi-smoothness in a neighborhood of random initialization for multi-layer networks?
• RQ4 Is there a finite-width equivalence between over-parameterized networks and the neural tangent kernel (NTK) similar to infinite-width results?
• RQ5 Do these results extend to CNNs and ResNets with ReLU activations and to various loss functions beyond squared loss?

Key findings

• Gradient descent finds an ε-error global minimum in poly(n,L,δ^{-1}) iterations for regression tasks, given width m ≥ poly(n,L,δ^{-1})·d.
• SGD achieves the same training-error objective in poly(n,L,δ^{-1})·log^2 m iterations with appropriate learning rate and mini-batch size.
• The objective near random initialization is almost convex and semi-smooth, precluding problematic saddles and enabling guaranteed descent.
• There is a polynomial-width equivalence between over-parameterized networks and the NTK in the finite-width setting (not only at infinite width).
• The analysis handles non-smooth ReLU activations and extends to CNNs and ResNets with broad applicability of the results.