[论文解读] Natasha 2: Faster Non-Convex Optimization Than SGD

Zeyuan Allen-Zhu|arXiv (Cornell University)|Aug 29, 2017

Advanced Bandit Algorithms Research被引用 53

一句话总结

Natasha 2 引入了一种在线随机方法，在光滑非凸优化中通过在 Oja 的算法实现的负曲率步与一阶更新之间交替，达到比 SGD 更快的收敛，在有利设置下实现 T = eO(1/ε3.25) 的近似局部极小点。

ABSTRACT

We design a stochastic algorithm to train any smooth neural network to $\varepsilon$-approximate local minima, using $O(\varepsilon^{-3.25})$ backpropagations. The best result was essentially $O(\varepsilon^{-4})$ by SGD. More broadly, it finds $\varepsilon$-approximate local minima of any smooth nonconvex function in rate $O(\varepsilon^{-3.25})$, with only oracle access to stochastic gradients.

研究动机与目标

• Motivate the design of online algorithms that beat SGD in finding ε-approximate local minima for smooth nonconvex objectives.
• Leverage bounded nonconvexity (σ) and negative-curvature directions to accelerate convergence.
• Develop Natasha1.5 and Natasha2 to exploit curvature information online, without full gradients or Hessians.
• Provide theoretical guarantees and asymptotic gradient (and Hessian) complexity improvements over prior online methods.

提出的方法

• Introduce Natasha1.5 (online variant of Natasha1) using a retraction term to stabilize updates and exploit σ-bounded nonconvexity.
• Combine Natasha1.5 with Oja’s online algorithm to perform negative curvature steps when a saddle point is detected.
• Formally define Natasha2 by alternating between escaping via negative curvature and safely reducing the objective using Natasha1.5 on a modified function.
• Prove convergence to ε-approximate stationary points and (ε, δ)-approximate local minima under standard smoothness and σ-bounded nonconvexity assumptions.
• Provide a proximal extension to minimize F(x)=ψ(x)+f(x) with a convex ψ.
• Compare online rates with existing SGD/SCSG/NEON-based methods.

实验结果

研究问题

• RQ1 Can online stochastic methods exploit σ-bounded nonconvexity to accelerate convergence beyond SGD?
• RQ2 Is it possible to reliably escape saddle points online by combining negative-curvature directions with first-order updates?
• RQ3 How can one design an online algorithm that alternates between escaping saddle points and converging to approximate local minima with provable guarantees?
• RQ4 What are the gradient and (where applicable) Hessian-vector product complexities for such online schemes compared to existing methods?

主要发现

• Natasha1.5 achieves an online gradient complexity T = Θ(L2/3 σ1/3 ε10/3) under σ-bounded nonconvexity and smoothness, improving over prior online rates.
• Natasha2 combines Oja’s online eigenvector finder with Natasha1.5 to obtain an online algorithm that finds an ε-approximate local minimum with ∥∇f(x)∥ ≤ ε and ∇2f(x) ⪰ −δI in T = eO(1/δ5 + 1/(δ ε3) + 1/ε3.25).
• Corollaries show T = eO(ε−3.25) for (ε, ε1/4)-approximate local minima and T = eO(ε−3.5) for (ε, ε1/2)-approximate local minima, surpassing several prior online methods.
• Natasha2 is capable of being implemented as a pure first-order method in follow-up work by replacing Hessian-vector products with gradient differences, while preserving convergence guarantees.
• The framework clarifies how to swing by saddle points by leveraging negative curvature directions and controlled perturbations without requiring exact full gradients or Hessian computations.