[Paper Review] An Improved Analysis of Training Over-parameterized Deep Neural Networks

Difan Zou, Quanquan Gu|arXiv (Cornell University)|Jun 11, 2019

Stochastic Gradient Optimization Techniques25 references47 citations

TL;DR

This paper provides a milder over-parameterization condition and faster convergence rates for GD/SGD training of deep ReLU networks, improving upon prior bounds especially for two-layer nets.

ABSTRACT

A recent line of research has shown that gradient-based algorithms with random initialization can converge to the global minima of the training loss for over-parameterized (i.e., sufficiently wide) deep neural networks. However, the condition on the width of the neural network to ensure the global convergence is very stringent, which is often a high-degree polynomial in the training sample size $n$ (e.g., $O(n^{24})$). In this paper, we provide an improved analysis of the global convergence of (stochastic) gradient descent for training deep neural networks, which only requires a milder over-parameterization condition than previous work in terms of the training sample size and other problem-dependent parameters. The main technical contributions of our analysis include (a) a tighter gradient lower bound that leads to a faster convergence of the algorithm, and (b) a sharper characterization of the trajectory length of the algorithm. By specializing our result to two-layer (i.e., one-hidden-layer) neural networks, it also provides a milder over-parameterization condition than the best-known result in prior work.

Motivation & Objective

• Motivate why over-parameterization enables global convergence of gradient-based training for deep ReLU nets.
• Derive tighter gradient lower bounds to achieve faster convergence rates.
• Sharpen trajectory length analysis to better characterize optimization dynamics.
• Specialize results to two-layer networks to compare with prior best-known bounds.
• Extend results to SGD and contrast with existing GD/SGD guarantees.

Proposed method

• Analyze training with Gaussian random initialization across all layers.
• Establish a gradient lower bound by introducing gradient regions almost orthogonal to layer outputs.
• Prove a sharper trajectory length bound to enable telescope-sum arguments.
• Derive conditions on width m that ensure convergence within a target accuracy ε.
• Specialize the general results to two-layer networks for direct comparison with prior work.
• Provide extension to SGD with corresponding over-parameterization and iteration bounds.

Experimental results

Research questions

• RQ1 Under what milder over-parameterization conditions can GD/SGD achieve ε-training loss for deep ReLU networks?
• RQ2 How do tighter gradient bounds and trajectory-length analyses affect iteration complexity compared to prior results?
• RQ3 What are the implications of these results for two-layer ReLU networks versus deeper architectures?
• RQ4 Can the results be extended to SGD and other loss functions beyond square loss?

Key findings

• GD achieves ε-training loss within O(n^2 L^2 log(1/ε)/φ) iterations under m = Ω(kn^8 L^12 φ^{-4} log^3(m)) hidden nodes per layer.
• SGD achieves ε-expected training loss within O(n^5 log(m) log^2(1/ε)/(Bφ^2)) iterations under m = Ω(kn^{17} L^{12} log^3(m) B^{-4} φ^{-8}).
• For two-layer networks, GD achieves ε-training loss within O(n^2 log(1/ε)/φ) iterations with m = Ω(kn^8 log^3(m)/φ^4).
• The over-parameterization condition improves over prior work by factors of at least n^4/φ and n^7 B^5 in respective settings.
• The analysis introduces gradient regions to obtain a tighter gradient lower bound, improving convergence rates.
• The trajectory-length analysis yields tighter bounds, enabling a more favorable dependence on problem parameters.