[Paper Review] Sharp Minima Can Generalize For Deep Nets

Laurent Dinh, Razvan Pascanu+2 more|arXiv (Cornell University)|Mar 15, 2017

Stochastic Gradient Optimization TechniquesComputer Science51 references148 citations

TL;DR

The paper argues that conventional notions of flatness/sharpness of minima are insufficient to explain generalization in deep nets due to non-identifiability and reparametrization; it demonstrates how equivalent models can be observed as arbitrarily sharp or flat minima, challenging flat-minima-based explanations.

ABSTRACT

Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.

Motivation & Objective

• Assess the validity of flatness-based explanations for generalization in deep rectified networks.
• Show how parameter space geometry and observational equivalence affect flatness measures.
• Demonstrate that reparametrizations can drastically alter the perceived sharpness without changing generalization.

Proposed method

• Define and compare multiple flatness/sharpness metrics (volume-based, Hessian-based, and epsilon-sharpness).
• Expose non-identifiability in rectified networks via non-negative homogeneity and alpha-scale transformations.
• Prove that alpha-scale transformations create infinitely large flat regions around minima or arbitrarily large Hessian spectra without changing predictions.
• Generalize transformations to multi-layer networks and derive implications for Hessian eigenvalues via D_alpha.

Experimental results

Research questions

• RQ1 Do standard flatness notions reliably predict generalization in deep rectified networks?
• RQ2 Can observationally equivalent parameterizations yield arbitrarily sharp or flat minima under common metrics?
• RQ3 How do reparametrizations affect the relationship between minimum geometry and generalization?
• RQ4 What are the limits of using Hessian-based measures as proxies for generalization in deep nets?

Key findings

• Volume-based epsilon-flatness can be infinite around typical one-hidden-layer rectified nets, meaning all minima appear equally flat under that metric.
• Hessian-based measures like spectral norm and trace can be arbitrarily manipulated via alpha-scale transformations without changing the function.
• There exist observationally equivalent minima with arbitrarily large Hessian eigenvalues, challenging the link between sharpness and generalization.
• For deeper networks, a subset of Hessian eigenvalues can be made large through parameter scaling, indicating sharp directions can exist without affecting generalization.
• Reparametrizations (including weight normalization and batch normalization variants) can alter the perceived geometry of minima, leaving generalization unaffected.