[Paper Review] Uniform convergence may be unable to explain generalization in deep learning

Vaishnavh Nagarajan, J. Zico Kolter|arXiv (Cornell University)|Feb 13, 2019

Stochastic Gradient Optimization TechniquesComputer Science39 references42 citations

TL;DR

The paper argues that uniform-convergence-based generalization bounds can be vacuous for overparameterized models trained by gradient descent, and demonstrates both empirical and theoretical failures even in algorithm-dependent settings.

ABSTRACT

Aimed at explaining the surprisingly good generalization behavior of overparameterized deep networks, recent works have developed a variety of generalization bounds for deep learning, all based on the fundamental learning-theoretic technique of uniform convergence. While it is well-known that many of these existing bounds are numerically large, through numerous experiments, we bring to light a more concerning aspect of these bounds: in practice, these bounds can {\em increase} with the training dataset size. Guided by our observations, we then present examples of overparameterized linear classifiers and neural networks trained by gradient descent (GD) where uniform convergence provably cannot "explain generalization" -- even if we take into account the implicit bias of GD {\em to the fullest extent possible}. More precisely, even if we consider only the set of classifiers output by GD, which have test errors less than some small $ε$ in our settings, we show that applying (two-sided) uniform convergence on this set of classifiers will yield only a vacuous generalization guarantee larger than $1-ε$. Through these findings, we cast doubt on the power of uniform convergence-based generalization bounds to provide a complete picture of why overparameterized deep networks generalize well.

Motivation & Objective

• Motivate why overparameterized deep networks generalize well beyond classic uniform convergence explanations.
• Empirically show that common uniform convergence-based bounds grow with training set size rather than decrease as expected.
• Provide theoretical constructions where two-sided uniform convergence bounds cannot explain generalization, even when considering the implicit bias of gradient descent.
• Highlight fundamental limitations of uniform convergence as a tool for understanding generalization in deep learning.

Proposed method

• Empirical analysis of weight norms and generalization on fully connected networks (depth 5, width 1024) trained on MNIST with SGD (lr=0.1, batch size 1) until 99% accuracy with margin gamma* = 10.
• Observation of how distance from initialization and product of spectral norms grow with data size m (at least m^0.4 and m, respectively).
• Evaluation of existing generalization bounds from prior work, showing bounds grow as Ω(m^0.68) due to numerator terms increasing with m.
• Theoretical construction of high-dimensional linear classifiers and neural networks trained by gradient descent where two-sided uniform convergence is provably vacuous.
• Definition and use of tightest algorithm-dependent uniform convergence (on the hypothesis set explored by the algorithm) to argue the limits of uniform convergence in explaining generalization.

Experimental results

Research questions

• RQ1 Can uniform convergence bounds provide non-vacuous generalization guarantees for overparameterized models trained by gradient descent?
• RQ2 Do weight-norm based quantities used in many bounds decrease with training set size in practice, aligning with observed generalization performance?
• RQ3 Are algorithm-dependent (tightest) uniform convergence bounds still vacuous for realistic deep learning settings?
• RQ4 What fundamental limitations does uniform convergence have in capturing the generalization behavior of overparameterized neural networks?

Key findings

• Weight norms such as distance from initialization and the product of spectral norms increase with training set size m (polynomial growth: at least m^0.4 and m, respectively).
• Generalization test error decreases with m (approximately 1/m^0.43 in a specific setup), but the corresponding bounds' numerator terms grow with m, causing bounds to increase (Ω(m^0.68)).
• Even pruning to the smallest hypothesis class explored by the algorithm (tightest uniform convergence) yields nearly vacuous generalization guarantees (bound close to 1 for small ε).
• Two-sided uniform convergence bounds fail to explain generalization in overparameterized linear classifiers and neural networks trained by GD/SGD, even when incorporating implicit regularization.
• One-sided PAC-Bayes bounds, though conceptually different, also reduce to nearly vacuous guarantees in these settings.