[Paper Review] Theory IIIb: Generalization in Deep Networks

Tomaso Poggio, Qianli Liao+4 more|arXiv (Cornell University)|Jun 29, 2018

Neural Networks and Applications4 references14 citations

TL;DR

This paper resolves the 'no-overfitting' puzzle in deep neural networks by proving that gradient descent on exponential-type losses (e.g., cross-entropy) implicitly regularizes deep networks toward minimum norm solutions in each layer, even without explicit regularization. The key result is that deep networks converge to a max-margin-like solution in the separable case, explaining generalization despite overparametrization and large capacity.

ABSTRACT

A main puzzle of deep neural networks (DNNs) revolves around the apparent absence of "overfitting", defined in this paper as follows: the expected error does not get worse when increasing the number of neurons or of iterations of gradient descent. This is surprising because of the large capacity demonstrated by DNNs to fit randomly labeled data and the absence of explicit regularization. Recent results by Srebro et al. provide a satisfying solution of the puzzle for linear networks used in binary classification. They prove that minimization of loss functions such as the logistic, the cross-entropy and the exp-loss yields asymptotic, "slow" convergence to the maximum margin solution for linearly separable datasets, independently of the initial conditions. Here we prove a similar result for nonlinear multilayer DNNs near zero minima of the empirical loss. The result holds for exponential-type losses but not for the square loss. In particular, we prove that the weight matrix at each layer of a deep network converges to a minimum norm solution up to a scale factor (in the separable case). Our analysis of the dynamical system corresponding to gradient descent of a multilayer network suggests a simple criterion for ranking the generalization performance of different zero minimizers of the empirical loss.

Motivation & Objective

• To resolve the longstanding puzzle of why deep neural networks generalize well despite high capacity and lack of explicit regularization.
• To extend the theoretical understanding of implicit regularization from linear networks to nonlinear deep networks.
• To characterize the convergence behavior of gradient descent in deep networks trained with exponential-type losses.
• To provide a criterion for ranking generalization performance across different zero-minimizers of the empirical risk.

Proposed method

• Analyzes the continuous-time gradient flow dynamics of deep networks with ReLU activations and exponential losses (e.g., cross-entropy).
• Uses positive homogeneity of ReLU to decompose weight matrices into norm and direction components, showing convergence to minimum norm solutions up to scale.
• Applies dynamical systems theory to show that gradient descent converges to a solution with minimal Frobenius norm per layer in the separable case.
• Derives the gradient dynamics of the cross-entropy loss and shows equivalence to exponential loss dynamics under normalization, enabling extension of linear network results to nonlinear settings.
• Considers weight normalization and discretization effects, showing that time-dependent learning rates are crucial for convergence in discrete GD.
• Demonstrates that misclassification errors lead to hyperbolic equilibria, implying stable but non-regularizing solutions only in the span of misclassified examples.

Experimental results

Research questions

• RQ1 Why do overparametrized deep networks not overfit despite fitting random labels?
• RQ2 Does gradient descent on exponential losses in deep networks implicitly regularize toward minimum norm solutions?
• RQ3 Can the generalization performance of different zero-minimizers of the empirical risk be ranked using a theoretical criterion?
• RQ4 How do the dynamics of gradient descent on deep networks with exponential losses compare to those in linear networks?
• RQ5 What role does weight normalization or learning rate scheduling play in the convergence and generalization of deep networks?

Key findings

• Gradient descent on deep networks with exponential-type losses converges to a minimum norm solution in each layer, up to a scale factor, in the separable case.
• The convergence is independent of initial conditions and corresponds to a max-margin-like solution, providing implicit regularization.
• The generalization error remains stable even when the number of parameters exceeds the training set size, due to this implicit regularization.
• The cross-entropy loss dynamics are equivalent to those of the exponential loss under normalization, allowing extension of linear network results to nonlinear deep networks.
• Misclassification errors do not regularize all weight components uniformly; only those in the span of misclassified examples are affected, leading to stable but non-regularizing equilibria.
• Discretization effects and time-dependent learning rates are critical for convergence, as they suppress noise and ensure asymptotic convergence to the minimum.