[Paper Review] Essentially No Barriers in Neural Network Energy Landscape

Felix Draxler, Kambis Veschgini+2 more|arXiv (Cornell University)|Mar 2, 2018

Stochastic Gradient Optimization Techniques21 references131 citations

TL;DR

The paper shows that minima of modern neural networks on CIFAR10/100 are connected by essentially flat paths with no significant energy barriers, suggesting minima form a single connected low-loss manifold.

ABSTRACT

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

Motivation & Objective

• Propose that neural network loss minima form a connected low-loss manifold rather than isolated points.
• Develop and apply a method to find minimum-energy paths between minima across architectures.
• Quantify barriers along paths between minima for state-of-the-art networks on CIFAR10 and CIFAR100.
• Provide qualitative explanation and empirical evidence for the observed path connectivity and low barriers.

Proposed method

• Define the minimum energy path (MEP) between two minima as the path minimizing the maximum loss along the path.
• Adopt and extend the Nudged Elastic Band (NEB) method to neural network loss landscapes to deform a path into a low-barrier trajectory.
• Use Automated Nudged Elastic Band (AutoNEB) to iteratively sample and refine paths with additional pivots where loss is high.
• Connect multiple minima across architectures (CNNs, ResNets, DenseNets) on CIFAR10/100 and evaluate the saddle points along the paths.
• Construct a minimum spanning tree over minima to summarize upper bounds on saddle energies between all pairs.
• Provide qualitative and quantitative analysis of the trajectory of network parameters along the MEPs.

Experimental results

Research questions

• RQ1 Do minima of modern neural networks lie in isolated basins or can they be connected by low-loss paths?
• RQ2 Can NEB/AutoNEB reveal minimum-energy paths that connect independent minima across architectures?
• RQ3 How do saddle point energies along these paths compare to the minima losses on training and test sets?
• RQ4 Does increasing depth and width reduce barrier heights between minima across CIFAR datasets?
• RQ5 What physical-chemistry inspired insights (e.g., resilience, redundancy) help explain observed path connectivity?

Key findings

• There exist continuous paths between independently trained minima where the training loss along the path remains essentially close to the minima values.
• Test loss along these paths remains essentially constant while test error increases only slightly.
• Barriers between minima decrease as architectures become deeper and wider, with barriers nearly vanishing for deep ResNets and DenseNets on CIFAR10/100.
• Local minimum energy paths (MEPs) show parameter trajectories that are smooth and deviate only moderately from linear interpolations, especially near saddle points.
• AutoNEB enables constructing low-barrier paths and provides upper bounds on saddle energies via a minimum spanning tree of minima.
• The study suggests minima lie on a single connected low-loss manifold rather than in isolated valleys.