[论文解读] NEON+: Accelerated Gradient Methods for Extracting Negative Curvature for Non-Convex Optimization

Yi Xu, Rong Jin+1 more|arXiv (Cornell University)|Dec 4, 2017

Stochastic Gradient Optimization Techniques参考文献 16被引用 8

一句话总结

本文提出 NEON+，一种利用 Nesterov 插值动量加速梯度法，可高效提取非凸优化中的负曲率，从而更快地逃离鞍点。其达到接近最优的迭代复杂度 Õ(1/ε^1.75)，以收敛至二阶平稳点，优于标准梯度下降法，并与目前已知最优的无 Hessian 方法相当。

ABSTRACT

Accelerated gradient (AG) methods are breakthroughs in convex optimization, improving the convergence rate of the gradient descent method for optimization with smooth functions. However, the analysis of AG methods for non-convex optimization is still limited. It remains an open question whether AG methods from convex optimization can accelerate the convergence of the gradient descent method for finding local minimum of non-convex optimization problems. This paper provides an affirmative answer to this question. In particular, we analyze two renowned variants of AG methods (namely Polyak's Heavy Ball method and Nesterov's Accelerated Gradient method) for extracting the negative curvature from random noise, which is central to escaping from saddle points. By leveraging the proposed AG methods for extracting the negative curvature, we present a new AG algorithm with double loops for non-convex optimization~\footnote{this is in contrast to a single-loop AG algorithm proposed in a recent manuscript~\citep{AGNON}, which directly analyzed the Nesterov's AG method for non-convex optimization and appeared online on November 29, 2017. However, we emphasize that our work is an independent work, which is inspired by our earlier work~\citep{NEON17} and is based on a different novel analysis.}, which converges to second-order stationary point $\x$ such that $\| abla f(\x)\|\leq ε$ and $ abla^2 f(\x)\geq -\sqrtε I$ with $\widetilde O(1/ε^{1.75})$ iteration complexity, improving that of gradient descent method by a factor of $ε^{-0.25}$ and matching the best iteration complexity of second-order Hessian-free methods for non-convex optimization.

研究动机与目标

• 解决凸优化中加速梯度方法是否可加速非凸设置收敛的开放问题。
• 开发一种一阶算法，高效识别并利用负曲率，以逃离鞍点。
• 实现比标准梯度下降更快的二阶平稳点收敛速率。
• 仅使用梯度信息，达到与无 Hessian 二阶方法最优已知迭代复杂度相当的收敛性能。

提出的方法

• 该方法采用 Nesterov 加速梯度（NAG）框架，从 Hessian 近似中的随机噪声中提取负曲率。
• 引入双层循环结构：内层循环利用 NAG 提取负曲率，外层循环将迭代点更新至二阶平稳点。
• 算法采用 Hessian-Lipschitz 假设，以限制曲率变化并确保收敛稳定性。
• 通过分析 Hessian-向量乘积的 NAG 动力学，提取负曲率方向，该动力学可增强负曲率方向的响应。
• 该方法避免显式计算 Hessian 矩阵，仅依赖梯度评估与随机噪声注入。
• 在光滑性与 Lipschitz Hessian 条件下分析收敛性，确保对非凸性的鲁棒性。

实验结果

研究问题

• RQ1 来自凸优化的加速梯度方法能否被适配以加速非凸问题的收敛？
• RQ2 Nesterov 的动量能否被有效用于提取负曲率以实现鞍点逃离？
• RQ3 一阶方法在非凸优化中收敛至二阶平稳点的最优迭代复杂度是多少？
• RQ4 能否在不计算 Hessian 矩阵的前提下，将收敛速率提升至超过标准梯度下降？

主要发现

• NEON+ 实现了 Õ(1/ε^1.75) 的迭代复杂度以收敛至二阶平稳点，其中 ∥∇f(x)∥ ≤ ε 且 ∇²f(x) ≥ −√εI。
• 该复杂度相比标准梯度下降法提升了 ε^−0.25 的因子，显著加速了收敛速度。
• 尽管仅使用一阶信息，该方法与目前已知最优的无 Hessian 二阶方法的迭代复杂度相当。
• 该算法仅通过梯度评估与随机噪声成功提取了负曲率，避免了显式 Hessian 矩阵计算。