[Paper Review] Saving Gradient and Negative Curvature Computations: Finding Local Minima More Efficiently

Yaodong Yu, Difan Zou+1 more|arXiv (Cornell University)|Dec 11, 2017

Stochastic Gradient Optimization Techniques39 references6 citations

TL;DR

This paper proposes a family of nonconvex optimization algorithms that significantly reduce gradient and negative curvature computations by dividing the search space into large and small gradient regions. The algorithm performs gradient descent only in large gradient regions and uses a single negative curvature step to escape small gradient regions, achieving improved runtime complexity and outperforming state-of-the-art methods in both deterministic and stochastic settings, especially in the finite-sum regime.

ABSTRACT

We propose a family of nonconvex optimization algorithms that are able to save gradient and negative curvature computations to a large extent, and are guaranteed to find an approximate local minimum with improved runtime complexity. At the core of our algorithms is the division of the entire domain of the objective function into small and large gradient regions: our algorithms only perform gradient descent based procedure in the large gradient region, and only perform negative curvature descent in the small gradient region. Our novel analysis shows that the proposed algorithms can escape the small gradient region in only one negative curvature descent step whenever they enter it, and thus they only need to perform at most $N_ε$ negative curvature direction computations, where $N_ε$ is the number of times the algorithms enter small gradient regions. For both deterministic and stochastic settings, we show that the proposed algorithms can potentially beat the state-of-the-art local minima finding algorithms. For the finite-sum setting, our algorithm can also outperform the best algorithm in a certain regime.

Motivation & Objective

• To design practical nonconvex optimization algorithms that minimize expensive gradient and negative curvature computations.
• To achieve faster convergence to approximate local minima by exploiting region-specific descent strategies.
• To improve runtime complexity over existing second-order and first-order methods for finding approximate second-order stationary points.
• To demonstrate superiority in both stochastic and finite-sum nonconvex optimization settings.
• To show that a single negative curvature step suffices to escape small gradient regions, drastically reducing computational overhead.

Proposed method

• The algorithm partitions the domain into large gradient regions (∥∇f(x)∥₂ > ϵ) and small gradient regions (∥∇f(x)∥₂ ≤ ϵ).
• In large gradient regions, the algorithm performs standard gradient descent or its variants.
• In small gradient regions, only one negative curvature descent step is applied to escape, ensuring efficient escape without repeated computation.
• The method leverages a novel analysis showing that one negative curvature step is sufficient to exit small gradient regions, limiting negative curvature computations to at most Nϵ times, where Nϵ is the number of entries into small gradient regions.
• The algorithm uses stochastic first-order oracles (e.g., SCSG, SVRG) in the large gradient region and one-step negative curvature updates in the small gradient region.
• Theoretical guarantees are derived using Markov inequality and concentration bounds to control error propagation and success probability.

Experimental results

Research questions

• RQ1 Can we design a nonconvex optimization algorithm that drastically reduces gradient and negative curvature computations while still guaranteeing convergence to an approximate local minimum?
• RQ2 Is it possible to achieve improved runtime complexity by limiting negative curvature computations to only one step per entry into a small gradient region?
• RQ3 How does the proposed method compare to state-of-the-art algorithms in terms of convergence rate and computational cost in stochastic and finite-sum settings?
• RQ4 Can the algorithm maintain high success probability while minimizing expensive Hessian or Hessian-vector product evaluations?
• RQ5 Does the region-based strategy—gradient descent in large gradient regions and one-step negative curvature in small ones—lead to provably better complexity bounds?

Key findings

• The proposed algorithm achieves runtime complexity of eO(ρ²∆f/ϵ³_H + min{ρ²∆f/ϵ³_H, Nϵ}) in the stochastic setting, improving upon existing methods.
• In the finite-sum setting, the algorithm achieves eO(L∆fn²/³/ϵ² + (n + n³/⁴L¹/²/ϵ_H¹/²)min{ρ²∆f/ϵ³_H, Nϵ}) runtime complexity, outperforming the best-known algorithms in certain regimes.
• The number of negative curvature direction computations is bounded by Nϵ, the number of times the algorithm enters small gradient regions, and only one such step is needed per entry.
• The algorithm guarantees escape from saddle points in a single negative curvature step, enabling efficient traversal of small gradient regions.
• Success probability is maintained at a constant level (≥2/3) by controlling error terms via Markov inequality and geometric distribution bounds on inner loop iterations.
• The method reduces reliance on expensive Hessian or Hessian-vector products, making it more practical than cubic regularization or iterative Hessian-based methods.