[Paper Review] Identifying and attacking the saddle point problem in high-dimensional non-convex optimization

Yann Dauphin, Razvan Pascanu+4 more|arXiv (Cornell University)|Jun 10, 2014

Sparse and Compressive Sensing Techniques20 references461 citations

TL;DR

This paper identifies saddle points—not local minima—as the primary obstacle in high-dimensional non-convex optimization, proposing the saddle-free Newton method to rapidly escape them by rescaling gradients using the absolute inverse Hessian. The method outperforms gradient descent and quasi-Newton methods in training deep and recurrent neural networks, achieving lower test error and faster convergence on benchmark tasks.

ABSTRACT

A central challenge to many fields of science and engineering involves minimizing non-convex error functions over continuous, high dimensional spaces. Gradient descent or quasi-Newton methods are almost ubiquitously used to perform such minimizations, and it is often thought that a main source of difficulty for these local methods to find the global minimum is the proliferation of local minima with much higher error than the global minimum. Here we argue, based on results from statistical physics, random matrix theory, neural network theory, and empirical evidence, that a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima, especially in high dimensional problems of practical interest. Such saddle points are surrounded by high error plateaus that can dramatically slow down learning, and give the illusory impression of the existence of a local minimum. Motivated by these arguments, we propose a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods. We apply this algorithm to deep or recurrent neural network training, and provide numerical evidence for its superior optimization performance.

Motivation & Objective

• To challenge the conventional belief that local minima are the main obstacle in high-dimensional non-convex optimization.
• To demonstrate that saddle points, not local minima, are the dominant barrier due to their exponential proliferation in high dimensions.
• To develop a second-order optimization method that efficiently escapes saddle points by leveraging curvature information differently than traditional quasi-Newton methods.
• To empirically validate the theoretical predictions about the correlation between critical point index and error level in neural network loss surfaces.
• To show that the saddle-free Newton method improves optimization performance on deep and recurrent neural networks compared to standard methods.

Proposed method

• Proposes the saddle-free Newton method, which rescales gradient steps by the absolute value of the inverse Hessian to escape saddle points.
• Uses a generalized trust region framework that defines the shape of the trust region based on curvature, rather than approximating the loss function to second order.
• Employs numerical techniques to search for critical points of any index (fraction of negative Hessian eigenvalues) in neural network error surfaces.
• Applies the method to train deep and recurrent neural networks, switching from SGD to the saddle-free Newton method once training stalls.
• Uses truncated Krylov subspace methods to approximate Hessian-vector products in high-dimensional settings where full Hessian computation is infeasible.
• Implements a trust-region approach that maintains stability and efficiency even far from saddle points.

Experimental results

Research questions

• RQ1 Are saddle points more prevalent than local minima in high-dimensional non-convex optimization problems?
• RQ2 Does the presence of high-error saddle points with large curvature plateaus significantly slow down gradient-based optimization?
• RQ3 Can a second-order optimization method be designed to rapidly escape saddle points by exploiting curvature information differently than quasi-Newton methods?
• RQ4 Is there a strong positive correlation between the index (fraction of negative Hessian eigenvalues) and the error level at critical points in neural network loss surfaces?
• RQ5 Does the saddle-free Newton method achieve faster convergence and better generalization than standard optimization algorithms in deep and recurrent neural networks?

Key findings

• Saddle points are exponentially more prevalent than local minima in high-dimensional non-convex optimization, especially at high error levels.
• Critical points with high error are overwhelmingly saddle points, with the fraction of negative curvature directions increasing with error, consistent with random matrix theory predictions.
• The saddle-free Newton method achieved a test error of 0.57 on the MNIST dataset, outperforming the previous best of 0.69 achieved by the Hessian-Free method.
• On character-level language modeling with a recurrent neural network, switching from SGD to the saddle-free Newton method led to a significant drop in error, indicating escape from a plateau around a saddle point.
• The solution found by the saddle-free Newton method had fewer negative eigenvalues than the SGD solution, confirming its ability to escape high-error saddle regions.
• The truncated Newton method with damping failed to improve performance after SGD stalled, highlighting the superiority of the proposed method in navigating saddle point plateaus.