[Paper Review] Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

Lei Wu, Zhanxing Zhu+1 more|arXiv (Cornell University)|Jun 30, 2017

Stochastic Gradient Optimization Techniques19 references126 citations

TL;DR

The paper argues that generalization in deep learning is mainly due to loss-landscape geometry, where good minima occupy large basins, causing random initialization to land in them; it provides theory for 2-layer nets and extensive empirical evidence for deeper nets.

ABSTRACT

It is widely observed that deep learning models with learned parameters generalize well, even with much more model parameters than the number of training samples. We systematically investigate the underlying reasons why deep neural networks often generalize well, and reveal the difference between the minima (with the same training error) that generalize well and those they don't. We show that it is the characteristics the landscape of the loss function that explains the good generalization capability. For the landscape of loss function for deep networks, the volume of basin of attraction of good minima dominates over that of poor minima, which guarantees optimization methods with random initialization to converge to good minima. We theoretically justify our findings through analyzing 2-layer neural networks; and show that the low-complexity solutions have a small norm of Hessian matrix with respect to model parameters. For deeper networks, extensive numerical evidence helps to support our arguments.

Motivation & Objective

• Explain why deep neural networks generalize well despite over-parameterization.
• Differentiate good minima from poor minima with the same training error.
• Explain why optimization from random initialization tends to find good minima.
• Tie empirical observations to theoretical landscape properties of loss functions.

Proposed method

• Analyze loss landscapes using basin-of-attractor concepts from dynamical systems.
• Develop a Hessian-based metric to quantify basin volume and solution complexity.
• Theoretically analyze 2-layer networks to relate low-complexity solutions to small Hessian norms.
• Provide empirical evidence on deeper nets via Hessian spectra and approximate Hessian norms.
• Introduce an attack data setup to generate bad minima with the same training error but poor generalization.

Experimental results

Research questions

• RQ1 What properties distinguish good (well-generalizing) minima from bad minima with the same training error?
• RQ2 Why do optimization methods with random initialization almost surely converge to good minima in deep networks?
• RQ3 How does the loss landscape's geometry influence the prevalence of good basins over bad basins?
• RQ4 To what extent do initialization, optimization dynamics, and landscape structure contribute to generalization?

Key findings

• Good minima occupy large basins of attraction; the volume of these basins dominates over bad minima.
• Random initialization places parameters in the good basin with overwhelming probability, leading to convergence to well-generalizing solutions.
• Low-complexity solutions in 2-layer networks have small Hessian norms, indicating flat regions with large basins.
• SGD alone is not the sole cause of good generalization; landscape structure largely governs outcomes.
• Empirical Hessian spectrum analyses show good minima lie in wide valleys with many near-zero eigenvalues, while bad minima have larger eigenvalues and tighter valleys.
• Spectral estimates of Hessian correlate with generalization performance in experiments on small and large networks.