[Paper Review] Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks

Yuan Cao, Quanquan Gu|arXiv (Cornell University)|May 30, 2019

Stochastic Gradient Optimization Techniques41 references93 citations

TL;DR

The paper proves that wide and deep ReLU networks trained with SGD have a generalization bound that matches a neural tangent random feature reference class, with rate ~n^{-1/2} independent of network width, under data classifiability by the NTRF model and connects to NTK theory.

ABSTRACT

We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected $0$-$1$ loss of a wide enough ReLU network trained with stochastic gradient descent (SGD) and random initialization can be bounded by the training loss of a random feature model induced by the network gradient at initialization, which we call a neural tangent random feature (NTRF) model. For data distributions that can be classified by NTRF model with sufficiently small error, our result yields a generalization error bound in the order of $ ilde{\mathcal{O}}(n^{-1/2})$ that is independent of the network width. Our result is more general and sharper than many existing generalization error bounds for over-parameterized neural networks. In addition, we establish a strong connection between our generalization error bound and the neural tangent kernel (NTK) proposed in recent work.

Motivation & Objective

• Motivate and analyze generalization in the over-parameterized regime for deep ReLU networks trained with SGD.
• Introduce the neural tangent random feature (NTRF) reference class as a data-clarifiability measure.
• Provide algorithm-dependent generalization bounds that do not depend on network width.
• Establish connections between NTRF-based bounds and neural tangent kernel (NTK) theory.
• Show improved sample complexity bounds compared to prior two-layer results.

Proposed method

• Define a fully connected deep ReLU network with width m and depth L and Gaussian He initialization.
• Introduce the neural tangent random feature (NTRF) function class based on network gradients at initialization.
• Prove a cumulative loss bound for SGD near initialization using the almost-linear-in-parameters property.
• Derive an expected 0-1 error bound by online-to-batch conversion and relate it to the NTRF class.
• Connect the bound to the neural tangent kernel (NTK) via the NTK matrix Θ^(L) and provide a corollary with NTK-based bounds.

Experimental results

Research questions

• RQ1 Under what conditions can SGD-trained wide and deep ReLU networks achieve small generalization error?
• RQ2 How does the data classifiability with respect to the NTRF reference class affect generalization bounds?
• RQ3 What is the relationship between SGD generalization bounds and neural tangent kernel theory for deep networks?
• RQ4 Can the analysis extend beyond two-layer networks to deep architectures with all parameters trained?

Key findings

• The expected 0-1 error of SGD-trained wide ReLU networks can be bounded by the training loss over the NTRF reference class.
• When data can be classified by the NTRF model with small error, the bound scales as ~tilde{O}(n^{-1/2}) and is independent of network width.
• The bound ties to NTK, giving a kernel-like expression ~ tilde{O}(L * sqrt(y^T Θ^(L)^{-1} y / n)).
• The result extends beyond two-layer networks and yields sharper bounds than prior random-feature-based analyses.
• Corollary 3.10 provides an NTK-based bound that is independent of width and depends on the NTK matrix Θ^(L).
• The framework implies that if the target function has bounded NTK-induced RKHS norm, SGD generalizes well.