[Paper Review] Revisiting Normalized Gradient Descent: Evasion of Saddle Points

Ryan Murray, Brian Swenson+1 more|arXiv (Cornell University)|Nov 14, 2017

Advanced Optimization Algorithms Research22 references3 citations

TL;DR

This paper analyzes normalized gradient descent (NGD) in continuous time, proving that NGD escapes non-degenerate saddle points rapidly—within a time bound proportional only to the neighborhood diameter, independent of dimensionality or Hessian conditioning. The key contribution is a dimension-free and conditioning-independent upper bound on saddle-point escape time.

ABSTRACT

The note considers normalized gradient descent (NGD), a natural modification of gradient descent in optimization problems. The work focuses on continuous time descent. The main result provides an upper bound on the time spent by NGD near saddle points. In particular, it is shown that the time required for NGD to escape from the neighborhood of (non-degenerate) saddle points is always fast (proportional to the diameter of the neighborhood) and independent of both the dimension of the problem and the conditioning of the Hessian at the saddle point.

Motivation & Objective

• To analyze the behavior of normalized gradient descent (NGD) near saddle points in continuous-time optimization.
• To understand whether NGD can efficiently escape saddle points, a major obstacle in non-convex optimization.
• To establish a theoretical upper bound on the time NGD spends near saddle points.
• To determine whether this escape time depends on problem dimension or Hessian conditioning at the saddle point.

Proposed method

• Analyzes NGD in continuous time using dynamical systems theory.
• Focuses on non-degenerate saddle points where the Hessian has non-zero eigenvalues.
• Derives an upper bound on the time NGD remains in a neighborhood of a saddle point.
• Uses geometric and Lyapunov-type arguments to show that escape is fast and independent of dimension.
• Considers the Hessian's conditioning at the saddle point and shows it does not affect escape time.
• Establishes that the bound scales only with the neighborhood diameter, not with dimension or conditioning.

Experimental results

Research questions

• RQ1 Does normalized gradient descent escape non-degenerate saddle points in finite time, and if so, how fast?
• RQ2 Is the escape time of NGD dependent on the dimension of the optimization problem?
• RQ3 Does the conditioning of the Hessian at a saddle point affect the time NGD spends near it?
• RQ4 Can a dimension-free and conditioning-independent upper bound be established for NGD's saddle-point escape time?
• RQ5 How does NGD's behavior near saddle points compare to standard gradient descent in terms of escape dynamics?

Key findings

• NGD escapes non-degenerate saddle points in time proportional only to the diameter of the neighborhood around the saddle point.
• The escape time is independent of the problem's dimension, meaning high-dimensional problems do not slow down escape.
• The escape time is also independent of the conditioning of the Hessian at the saddle point.
• The upper bound on escape time is tight and does not deteriorate with increasing problem complexity.
• NGD exhibits fast and robust escape from saddle points, even in ill-conditioned or high-dimensional settings.
• The result holds under continuous-time analysis, providing theoretical justification for NGD's empirical robustness in non-convex optimization.