[Paper Review] An Empirical Analysis of Deep Network Loss Surfaces

Daniel Jiwoong Im, Michael Tao+1 more|arXiv (Cornell University)|Apr 24, 2017

Advanced Neural Network Applications19 references22 citations

TL;DR

This paper empirically investigates the geometry of loss surfaces in deep neural networks using visualization techniques on polygonal projections, revealing that stochastic optimization methods like SGD and Adam converge to minima with distinct structural properties. The key contribution is demonstrating that these minima are often connected by low-loss paths, suggesting a favorable optimization landscape despite non-convexity.

ABSTRACT

The training of deep neural networks is a high-dimension optimization problem with respect to the loss function of a model. Unfortunately, these functions are of high dimension and non-convex and hence difficult to characterize. In this paper, we empirically investigate the geometry of the loss functions for state-of-the-art networks with multiple stochastic optimization methods. We do this through several experiments that are visualized on polygons to understand how and when these stochastic optimization methods find minima.

Motivation & Objective

• To understand the geometric structure of loss surfaces in deep neural networks, which are high-dimensional and non-convex.
• To investigate how stochastic optimization methods such as SGD and Adam navigate these complex loss landscapes.
• To visualize and analyze the connectivity and curvature of minima in state-of-the-art deep networks.
• To determine whether minima found by different optimization methods are structurally similar or distinct.
• To assess the role of optimization dynamics in shaping the loss surface geometry.

Proposed method

• The authors use dimensionality reduction techniques to project high-dimensional loss surfaces onto 2D polygons for visualization.
• They train state-of-the-art deep networks using multiple stochastic optimization methods, including SGD and Adam.
• The loss values are computed along linear interpolation paths between found minima to assess connectivity.
• The geometry of the loss surface is analyzed by measuring curvature and loss variation across different regions.
• Experiments are conducted on standard benchmarks and network architectures to ensure generalizability.
• Visualization tools are used to compare the structural properties of minima found by different optimizers.

Experimental results

Research questions

• RQ1 Are the minima found by stochastic optimization methods in deep networks connected by low-loss paths?
• RQ2 How do the geometric properties of loss surfaces differ across optimization methods like SGD and Adam?
• RQ3 To what extent are minima in deep network loss surfaces connected, suggesting a favorable optimization landscape?
• RQ4 Do different optimization methods converge to minima with similar or distinct curvature and loss characteristics?
• RQ5 Can the loss surface geometry explain the generalization performance of trained models?

Key findings

• Minima found by different optimization methods are often connected by low-loss paths, indicating a highly connected and favorable optimization landscape.
• The loss surface exhibits a high degree of connectivity between minima, suggesting that optimization is not trapped in isolated local minima.
• The curvature of the loss surface near minima is generally low, supporting the existence of wide minima that generalize well.
• SGD tends to find flatter minima compared to Adam, which may explain its better generalization in some settings.
• The visualization of loss surfaces reveals that the loss function is less non-convex than previously assumed, with many minima lying on low-loss paths.
• The results suggest that the optimization process in deep networks is guided by a landscape that facilitates convergence to generalizable solutions.