[Paper Review] Gradient Descent Efficiently Finds the Cubic-Regularized Non-Convex Newton Step

Yair Carmon, John C. Duchi|arXiv (Cornell University)|Dec 2, 2016

Sparse and Compressive Sensing Techniques11 references58 citations

TL;DR

This paper demonstrates that gradient descent efficiently approximates the globally optimal solution for cubic-regularized non-convex Newton steps, achieving $\varepsilon$-accuracy in $O(\varepsilon^{-1}\log(1/\varepsilon))$ steps for large $\varepsilon$ and $O(\log(1/\varepsilon))$ steps for small $\varepsilon$, with logarithmic dependence on dimension. The result establishes a convergence rate to second-order stationary points for general smooth non-convex functions.

ABSTRACT

We consider the minimization of non-convex quadratic forms regularized by a cubic term, which exhibit multiple saddle points and poor local minima. Nonetheless, we prove that, under mild assumptions, gradient descent approximates the $ extit{global minimum}$ to within $\varepsilon$ accuracy in $O(\varepsilon^{-1}\log(1/\varepsilon))$ steps for large $\varepsilon$ and $O(\log(1/\varepsilon))$ steps for small $\varepsilon$ (compared to a condition number we define), with at most logarithmic dependence on the problem dimension. When we use gradient descent to approximate the Nesterov-Polyak cubic-regularized Newton step, our result implies a rate of convergence to second-order stationary points of general smooth non-convex functions.

Motivation & Objective

• To analyze the convergence of gradient descent for minimizing non-convex quadratic forms regularized by a cubic term.
• To establish convergence rates to the global minimum under mild assumptions.
• To demonstrate that gradient descent approximates the Nesterov-Polyak cubic-regularized Newton step with low dimension dependence.
• To derive a convergence rate to second-order stationary points for general smooth non-convex functions.

Proposed method

• Gradient descent is applied to minimize a non-convex quadratic function regularized by a cubic term.
• The analysis introduces a condition number to characterize the problem's difficulty and dependence on $\varepsilon$.
• Convergence bounds are derived using smoothness and curvature assumptions, with logarithmic dependence on dimension.
• The method leverages the structure of the cubic-regularized Newton step to bound the number of iterations required.
• Theoretical guarantees are established via iterative descent with error control in the objective gap.

Experimental results

Research questions

• RQ1 Can gradient descent efficiently approximate the global minimum of a cubic-regularized non-convex quadratic form?
• RQ2 What is the convergence rate of gradient descent to the global minimum in terms of $\varepsilon$ and the condition number?
• RQ3 How does the dimensionality affect the convergence complexity of gradient descent in this setting?
• RQ4 Does approximating the cubic-regularized Newton step via gradient descent yield a rate to second-order stationary points?

Key findings

• Gradient descent achieves $O(\varepsilon^{-1}\log(1/\varepsilon))$ convergence steps for large $\varepsilon$ to reach $\varepsilon$-accuracy in the global minimum.
• For small $\varepsilon$, the convergence rate improves to $O(\log(1/\varepsilon))$ steps, indicating faster convergence in favorable regimes.
• The convergence rate exhibits at most logarithmic dependence on the problem dimension, ensuring scalability.
• The method provides a theoretical foundation for using gradient descent to approximate the Nesterov-Polyak cubic-regularized Newton step.
• The result implies a convergence rate to second-order stationary points for general smooth non-convex functions.