[Paper Review] Asymmetric Valleys: Beyond Sharp and Flat Local Minima

Haowei He, Gao Huang+1 more|arXiv (Cornell University)|Feb 2, 2019

Stochastic Gradient Optimization Techniques43 references35 citations

TL;DR

The paper introduces asymmetric valleys in loss landscapes, proves that biased SGD solutions toward the flat side generalize better, and shows SGD averaging (e.g., SWA) naturally yields such biased solutions, with BN units contributing to asymmetry.

ABSTRACT

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good generalization performance. In this paper, we observe that local minima of modern deep networks are more than being flat or sharp. Specifically, at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side--we formally define such minima as asymmetric valleys. Under mild assumptions, we prove that for asymmetric valleys, a solution biased towards the flat side generalizes better than the exact minimizer. Further, we show that simply averaging the weights along the SGD trajectory gives rise to such biased solutions implicitly. This provides a theoretical explanation for the intriguing phenomenon observed by Izmailov et al. (2018). In addition, we empirically find that batch normalization (BN) appears to be a major cause for asymmetric valleys.

Motivation & Objective

• Motivate the existence of asymmetric loss valleys beyond flat/sharp minima in deep nets.
• Define asymmetric directions and valleys formally and establish their prevalence in modern architectures.
• Theoretically show that biased solutions toward the flat side yield better generalization under mild shifts between empirical and population loss.
• Demonstrate that averaging SGD iterates (including SWA) induces the desired bias toward the flat side.
• Investigate the role of batch normalization in shaping asymmetric valleys and the effectiveness of BN-targeted averaging.

Proposed method

• Formalize asymmetric directions and asymmetric valleys with precise definitions (Definition 1 and Definition 2).
• Introduce the (δ, R)-shift gap to relate population and empirical losses (Definition 3) and state two assumptions (Random shift, Locally asymmetric).
• Prove a bias/generalization theorem: under these assumptions, biased solutions toward the flat side generalize better than empirical minimizers (Theorem 1).
• Provide empirical verification of shift gaps and asymmetry in ResNet/ DenseNet on CIFAR datasets (Figures 3, 4).
• Show that averaging SGD iterates yields biased solutions toward the flat side, with theoretical support (Theorem 2) and empirical validation (Figures 5–9).
• Analyze SWA and large vs. small batch effects, clarifying that improved generalization arises from bias in asymmetric valleys rather than width alone (Figures 8–11).
• Examine the role of Batch Normalization in creating asymmetric directions and demonstrate BN-focused averaging benefits (Section 7).

Experimental results

Research questions

• RQ1 Do modern deep networks exhibit asymmetric valleys in their loss landscapes?
• RQ2 Does a solution biased toward the flat side of an asymmetric valley generalize better than the exact empirical minimizer?
• RQ3 Can SGD averaging (e.g., SWA) implicitly find such biased solutions?
• RQ4 What is the role of Batch Normalization in shaping asymmetry and in the effectiveness of averaging?
• RQ5 Are improvements from small/batch size training or SWA due to bias toward flat directions rather than pure flatness/width?

Key findings

• Asymmetric valleys are prevalent in loss landscapes of modern networks like ResNet and DenseNet on CIFAR datasets.
• A biased solution toward the flat side of an asymmetric valley generalizes better than the empirical minimizer under a calm shift between population and empirical losses (Theorem 1).
• A simple SGD trajectory average yields biased solutions toward the flat side, supporting a practical algorithmic path to better generalization (Theorem 2).
• Averaging SGD iterates in practice (e.g., SWA) produces solutions biased toward flat sides and yields better test performance despite sometimes higher training loss (Figure 8 and related discussion).
• Batch Normalization contributes to creating asymmetric directions, and averaging BN parameters can be particularly effective (SWA-BN outperforming SWA-Non-BN in BN-enabled networks).
• Visualizations show that in high-dimensional landscapes, different basins can appear wider/shallower due to asymmetry rather than true width differences (Section 6).