[Paper Review] An Alternative View: When Does SGD Escape Local Minima?

Robert Kleinberg, Yuanzhi Li+1 more|arXiv (Cornell University)|Feb 17, 2018

Stochastic Gradient Optimization Techniques9 references42 citations

TL;DR

The paper analyzes SGD as optimization on a convolved (smoothed) version of the loss, showing that if the convolved function is one-point strongly convex toward a target x* then SGD will approach and stay near x* with constant probability, explaining why SGD avoids sharp local minima.

ABSTRACT

Stochastic gradient descent (SGD) is widely used in machine learning. Although being commonly viewed as a fast but not accurate version of gradient descent (GD), it always finds better solutions than GD for modern neural networks. In order to understand this phenomenon, we take an alternative view that SGD is working on the convolved (thus smoothed) version of the loss function. We show that, even if the function $f$ has many bad local minima or saddle points, as long as for every point $x$, the weighted average of the gradients of its neighborhoods is one point convex with respect to the desired solution $x^*$, SGD will get close to, and then stay around $x^*$ with constant probability. More specifically, SGD will not get stuck at "sharp" local minima with small diameters, as long as the neighborhoods of these regions contain enough gradient information. The neighborhood size is controlled by step size and gradient noise. Our result identifies a set of functions that SGD provably works, which is much larger than the set of convex functions. Empirically, we observe that the loss surface of neural networks enjoys nice one point convexity properties locally, therefore our theorem helps explain why SGD works so well for neural networks.

Motivation & Objective

• Motivate and formalize an alternative view of SGD as operating on a convolved version of the loss function.
• Show that SGD can converge to a good local minimum even when f has many bad local minima or saddles, via one-point convexity of the convolved function.
• Identify realistic conditions under which SGD provably stays near a target solution and escapes sharp minima.

Proposed method

• Introduce convolved loss g_t(y)=E_{ω∈W(x)}[f(y−ηω)].
• Define analysis sequence y_t=x_t−η∇f(x_t) and relate SGD to gradient descent on g_t.
• State Assumption 1 (Main Assumption): f becomes c-one-point strongly convex after convolution with noise, relative to x*.
• Prove Theorem 4: with appropriate η, r (noise bound), and c, SGD keeps y_t within O(η r^2/c) of x* for a future time horizon with constant probability.
• Provide Corollary 2 (Shrinking Learning Rate): using a staged learning rate improves local convergence.
• Offer empirical observations that local loss surfaces exhibit one-point convexity properties and explain the role of step size.

Experimental results

Research questions

• RQ1 Under what conditions does SGD escape sharp local minima due to the smoothing effect of gradient noise?
• RQ2 Can one-point convexity of the convolved loss guarantee that SGD stays near a target solution, and what are the quantitative implications for step size and noise?
• RQ3 How does the learning rate schedule interact with the convolution-based view to influence convergence to good local minima?

Key findings

• SGD operates on a convolved (smoothed) version of the loss, which can eliminate sharp local minima.
• If the convolved loss is c-one-point strongly convex toward x*, SGD will converge toward x* and stay near it with constant probability.
• There is a quantifiable bound showing proximity to x* of order O(η r^2/c) within a time window, given η is suitably small and noise is bounded.
• Too large a step size can hinder convergence, even under one-point convexity of the original function, as full gradient descent may diverge away from x*.
• Empirically, modern neural networks exhibit local one-point convexity in their loss landscapes, supporting the proposed theory.