[Paper Review] On Noisy Negative Curvature Descent: Competing with Gradient Descent for Faster Non-convex Optimization

Mingrui Liu, Tianbao Yang|arXiv (Cornell University)|Sep 25, 2017

Sparse and Compressive Sensing Techniques20 references16 citations

TL;DR

This paper proposes a novel noisy negative curvature descent algorithm that reduces Hessian-vector product computations in non-convex optimization by requiring only approximate smallest eigen-vectors with precision tied to the gradient norm, not to a small absolute threshold. The method competes with gradient descent and achieves second-order stationary points with time complexity matching the best in literature, while enabling stochastic variants with sample-size-independent complexity.

ABSTRACT

The Hessian-vector product has been utilized to find a second-order stationary solution with strong complexity guarantee (e.g., almost linear time complexity in the problem's dimensionality). In this paper, we propose to further reduce the number of Hessian-vector products for faster non-convex optimization. Previous algorithms need to approximate the smallest eigen-value with a sufficient precision (e.g., $ε_2\ll 1$) in order to achieve a sufficiently accurate second-order stationary solution (i.e., $λ_{\min}( abla^2 f(\x))\geq -ε_2)$. In contrast, the proposed algorithms only need to compute the smallest eigen-vector approximating the corresponding eigen-value up to a small power of current gradient's norm. As a result, it can dramatically reduce the number of Hessian-vector products during the course of optimization before reaching first-order stationary points (e.g., saddle points). The key building block of the proposed algorithms is a novel updating step named the NCG step, which lets a noisy negative curvature descent compete with the gradient descent. We show that the worst-case time complexity of the proposed algorithms with their favorable prescribed accuracy requirements can match the best in literature for achieving a second-order stationary point but with an arguably smaller per-iteration cost. We also show that the proposed algorithms can benefit from inexact Hessian by developing their variants accepting inexact Hessian under a mild condition for achieving the same goal. Moreover, we develop a stochastic algorithm for a finite or infinite sum non-convex optimization problem. To the best of our knowledge, the proposed stochastic algorithm is the first one that converges to a second-order stationary point in {\it high probability} with a time complexity independent of the sample size and almost linear in dimensionality.

Motivation & Objective

• To reduce the number of Hessian-vector product computations required for finding second-order stationary points in non-convex optimization.
• To develop an algorithm that competes effectively with gradient descent in convergence speed while maintaining strong theoretical complexity guarantees.
• To enable the use of inexact or sub-sampled Hessians under mild conditions without sacrificing convergence to second-order stationary points.
• To design a stochastic algorithm for finite or infinite sum problems that achieves second-order stationarity with time complexity independent of sample size and almost linear in dimensionality.
• To provide theoretical guarantees for convergence to second-order stationary points under noisy negative curvature descent with reduced per-iteration cost.

Proposed method

• Introduces the NCG (Noisy Curvature Descent) step as a core update mechanism that leverages approximate smallest eigen-vectors of the Hessian to escape saddle points.
• Replaces the need for high-precision approximation of the smallest eigenvalue (e.g., ε₂ ≪ 1) with a requirement proportional to the current gradient norm’s power, significantly reducing Hessian-vector product calls.
• Develops variants that accept inexact Hessians (e.g., sub-sampled Hessian) under a mild condition, preserving convergence to second-order stationary points.
• Proposes a stochastic algorithm that computes only a sub-sampled gradient and a noisy negative curvature of a sub-sampled Hessian per iteration.
• Employs power method-like iterations to approximate the smallest eigen-vector with reduced accuracy requirements, lowering computational cost per iteration.
• Combines gradient descent and NCG steps adaptively, ensuring competitive convergence while minimizing expensive Hessian operations.

Experimental results

Research questions

• RQ1 Can we reduce the number of Hessian-vector product computations in non-convex optimization by relaxing the precision requirement on the smallest eigenvalue approximation?
• RQ2 Can a noisy negative curvature descent method outperform or match gradient descent in convergence speed while maintaining strong theoretical guarantees?
• RQ3 Is it possible to design a stochastic algorithm for finite or infinite sum problems that converges to a second-order stationary point with complexity independent of sample size?
• RQ4 Can inexact or sub-sampled Hessians be used effectively in second-order optimization without degrading convergence to second-order stationarity?
• RQ5 What is the minimal accuracy requirement on the smallest eigen-vector approximation to ensure convergence to a second-order stationary point?

Key findings

• The proposed algorithm matches the best-known time complexity in the literature for reaching a second-order stationary point, but with a potentially smaller per-iteration computational cost.
• The method reduces the required precision of the smallest eigenvalue approximation from ε₂ ≪ 1 to a small power of the current gradient norm, drastically cutting down Hessian-vector product usage.
• The stochastic variant computes only a sub-sampled gradient and a noisy negative curvature of a sub-sampled Hessian per iteration, achieving convergence with complexity independent of the sample size.
• The algorithm is the first known stochastic method to converge to a second-order stationary point in high probability with time complexity almost linear in the problem dimension and independent of sample size.
• Theoretical analysis shows that the proposed method with inexact Hessian maintains convergence to second-order stationarity under a mild condition on the Hessian approximation error.
• The NCG step enables effective competition with gradient descent by efficiently escaping saddle points with fewer Hessian-vector products.