[論文レビュー] Visualizing the Loss Landscape of Neural Nets

Hao Li, Zheng Xu+3 more|arXiv (Cornell University)|Dec 28, 2017

Advanced Neural Network Applications参考文献 31被引用数 621

ひとこと要約

この論文はニューラルネットワークの損失景観を可視化し、有意義な幾何学的比較を可能にするフィルタ別正規化を導入し、景観の形状をアーキテクチャ、学習パラメータ、一般化と結びつけ、最適化軌跡の可視化も行う。

ABSTRACT

Neural network training relies on our ability to find "good" minimizers of highly non-convex loss functions. It is well-known that certain network architecture designs (e.g., skip connections) produce loss functions that train easier, and well-chosen training parameters (batch size, learning rate, optimizer) produce minimizers that generalize better. However, the reasons for these differences, and their effects on the underlying loss landscape, are not well understood. In this paper, we explore the structure of neural loss functions, and the effect of loss landscapes on generalization, using a range of visualization methods. First, we introduce a simple "filter normalization" method that helps us visualize loss function curvature and make meaningful side-by-side comparisons between loss functions. Then, using a variety of visualizations, we explore how network architecture affects the loss landscape, and how training parameters affect the shape of minimizers.

研究の動機と目的

• Understand how neural loss landscapes relate to trainability and generalization.
• Develop a robust visualization method that accounts for scale-invariance in networks.
• Empirically characterize how architecture (skip connections, depth, width) affects landscape geometry.
• Examine how training parameters (batch size, weight decay) shape minimizers and their generalization.
• Visualize optimization trajectories to reveal their dimensionality and dynamics.

提案手法

• Propose filter-wise normalization to enable meaningful 2D/contour visualizations across architectures.
• Use high-resolution 2D contour plots around minimizers to study sharpness/flatness.
• Compute Hessian eigenvalues (min/max) via Lanczos to quantify non-convexity around minima.
• Visualize SGD trajectories with PCA-based directions to reveal low-dimensional structure.
• Compare architectures (ResNet variants, DenseNet, Wide-ResNet) and training settings on CIFAR-10.
• Provide code/plots resources for reproducibility.

実験結果

リサーチクエスチョン

• RQ1 How does network architecture (e.g., skip connections) shape the loss landscape and its non-convexity?
• RQ2 How do training parameters (batch size, weight decay) influence the sharpness of minimizers and generalization?
• RQ3 Is there a systematic relationship between the geometry of minima (flat vs sharp) and generalization performance?
• RQ4 Can loss-landscape visualizations reveal why certain architectures are easier to train than others?
• RQ5 What is the proper way to visualize optimization trajectories in high-dimensional spaces?

主な発見

• Filter normalization enables side-by-side comparisons of minimizers and reveals correlations between sharpness and generalization that are robust to architectural differences.
• Skip connections promote flat minimizers and suppress chaotic non-convexity as depth increases.
• Without skip connections, deeper nets show transitions from nearly convex to chaotic loss landscapes, correlating with worse generalization.
• Wider networks exhibit flatter minima and reduced non-convexity, with sharpness aligning with test error.
• Optimization trajectories are intrinsically low-dimensional, often captured by PCA directions, and visualization along these directions reveals descent dynamics.
• Hessian analysis shows that convex-looking regions have small negative eigenvalues, while chaotic regions exhibit larger negative curvature.