[Paper Review] Third-order Smoothness Helps: Even Faster Stochastic Optimization Algorithms for Finding Local Minima

Yaodong Yu, Pan Xu+1 more|arXiv (Cornell University)|Dec 18, 2017

Sparse and Compressive Sensing Techniques28 references3 citations

TL;DR

This paper proposes a novel stochastic optimization algorithm that leverages third-order smoothness to accelerate convergence to local minima in nonconvex problems. By exploiting higher-order smoothness, the algorithm achieves an improved gradient complexity of Õ(ϵ⁻¹⁰ᐟ³) stochastic gradient evaluations—representing a significant speedup over prior state-of-the-art methods that require Õ(ϵ⁻⁷ᐟ²) evaluations—while using only first-order oracles and minimizing expensive negative curvature computations.

ABSTRACT

We propose stochastic optimization algorithms that can find local minima faster than existing algorithms for nonconvex optimization problems, by exploiting the third-order smoothness to escape non-degenerate saddle points more efficiently. More specifically, the proposed algorithm only needs $ ilde{O}(ε^{-10/3})$ stochastic gradient evaluations to converge to an approximate local minimum $\mathbf{x}$, which satisfies $\| abla f(\mathbf{x})\|_2\leqε$ and $λ_{\min}( abla^2 f(\mathbf{x}))\geq -\sqrtε$ in the general stochastic optimization setting, where $ ilde{O}(\cdot)$ hides logarithm polynomial terms and constants. This improves upon the $ ilde{O}(ε^{-7/2})$ gradient complexity achieved by the state-of-the-art stochastic local minima finding algorithms by a factor of $ ilde{O}(ε^{-1/6})$. For nonconvex finite-sum optimization, our algorithm also outperforms the best known algorithms in a certain regime.

Motivation & Objective

• To design faster stochastic optimization algorithms that find local minima in nonconvex problems with reduced computational cost.
• To exploit third-order smoothness of the objective function to accelerate escape from non-degenerate saddle points.
• To achieve faster convergence than existing stochastic local minima finding algorithms while relying only on first-order oracles.
• To minimize the number of expensive negative curvature descent steps and gradient evaluations in high-dimensional, large-scale settings.

Proposed method

• The algorithm integrates first-order stochastic gradient descent with a novel negative curvature descent step that leverages third-order smoothness to enable larger step sizes.
• It uses a stochastic variant of the cubic regularization method, enhanced by third-order smoothness to improve convergence rates.
• The method dynamically switches between an SCSG (Stochastically Controlled Stochastic Gradient) epoch and a single negative curvature descent step per outer iteration.
• A new analysis framework is developed to bound the number of iterations where gradient norm is large, using concentration inequalities and third-order smoothness to control error terms.
• The algorithm employs online Oja’s algorithm and FastPCA/Neon2 for efficient computation of negative curvature directions.
• Theoretical analysis establishes convergence to (ϵ, ϵH)-second-order stationary points with improved gradient complexity.

Experimental results

Research questions

• RQ1 Can third-order smoothness of the objective function be exploited to accelerate escape from saddle points in stochastic nonconvex optimization?
• RQ2 What is the theoretical gradient complexity improvement when third-order smoothness is used in stochastic local minima finding algorithms?
• RQ3 How can first-order oracles be used more efficiently to approximate negative curvature directions without solving cubic subproblems?
• RQ4 Can the number of expensive negative curvature computations be reduced while maintaining fast convergence to local minima?

Key findings

• The proposed algorithm achieves a gradient complexity of Õ(ϵ⁻¹⁰ᐟ³) stochastic gradient evaluations to find an (ϵ, ϵH)-second-order stationary point, improving upon the state-of-the-art Õ(ϵ⁻⁷ᐟ²) complexity by a factor of Õ(ϵ⁻¹ᐟ⁶).
• For finite-sum problems, the algorithm requires Õ(n²ᐟ³ϵ⁻² + nϵH⁻² + n³ᐟ⁴ϵH⁻⁵ᐟ²) stochastic gradient evaluations, which outperforms existing methods in certain parameter regimes.
• The algorithm reduces the number of negative curvature descent steps per iteration to at most one, significantly lowering computational overhead compared to methods requiring multiple such steps.
• By exploiting third-order smoothness, the algorithm enables larger step sizes during negative curvature descent, leading to faster convergence and improved iteration efficiency.
• The theoretical analysis proves that the algorithm converges to a local minimum with high probability under standard smoothness and noise assumptions, with runtime complexity bounded by Õ(L³Δf/ϵH²) for the main loop and Õ(L²₁L³Δf/ϵH⁴) for the negative curvature computation.