[论文解读] How to Escape Saddle Points Efficiently

Chi Jin, Rong Ge+3 more|arXiv (Cornell University)|Mar 2, 2017

Sparse and Compressive Sensing Techniques参考文献 18被引用 231

一句话总结

本文表明扰动梯度下降在近似 dimenson-free 的迭代复杂度下找到一个 ε-二阶驻点（因此在严格鞍点下是局部极小），在多项对数因子内与一阶收敛速率相匹配。

ABSTRACT

This paper shows that a perturbed form of gradient descent converges to a second-order stationary point in a number iterations which depends only poly-logarithmically on dimension (i.e., it is almost "dimension-free"). The convergence rate of this procedure matches the well-known convergence rate of gradient descent to first-order stationary points, up to log factors. When all saddle points are non-degenerate, all second-order stationary points are local minima, and our result thus shows that perturbed gradient descent can escape saddle points almost for free. Our results can be directly applied to many machine learning applications, including deep learning. As a particular concrete example of such an application, we show that our results can be used directly to establish sharp global convergence rates for matrix factorization. Our results rely on a novel characterization of the geometry around saddle points, which may be of independent interest to the non-convex optimization community.

研究动机与目标

• Motivate the need to escape saddle points in non-convex optimization and improve training efficiency in high dimensions.
• Develop a gradient-descent-based method with perturbations that converges to second-order stationary points.
• Quantify iteration complexity and show near dimension-free rates under mild smoothness and Hessian-Lipschitz assumptions.
• Demonstrate applicability to problems like matrix factorization and discuss local structure benefits.

提出的方法

• 提出一个扰动梯度下降 (PGD) 元算法，当梯度较小时添加随机扰动。
• 在 ℓ-平滑和 ρ-Hessian Lipschitz 目标下分析 PGD，以界定达到 ε-二阶驻点所需的时间。
• 使用基于阈值的扰动调度，扰动从一个 d 维球中均匀抽取。
• 证明扰动通过围绕鞍点的几何“带状”论证使其能够逃离鞍点。
• 给出参数选择（步长 η = O(1/ℓ)、扰动半径 r、阈值），以实现主要保证。
• 将分析扩展到具有严格鞍点属性和局部强凸性的设定，以获得改进的收敛速率。

实验结果

研究问题

• RQ1 梯度下降在偶尔扰动的情况下，是否能够在多项式时间内逃离所有鞍点？
• RQ2 对于 ρ-Hessian Lipschitz 的函数，达到 ε-二阶驻点的迭代复杂度是多少？
• RQ3 局部几何结构（严格鞍点、局部强凸性）如何影响收敛速率？
• RQ4 该方法是否能够为矩阵分解等问题提供全局收敛保证？

主要发现

• 扰动梯度下降在 Õ(ℓ(f(x0)−f*)/ε^2) 次迭代内达到 ε-二阶驻点，至多带有 polylog(d) 因子。
• 在严格鞍点假设下，该方法在同样的复杂度界内找到局部极小，去除对数因子后。
• 在局部强凸性下，第二阶段的收敛改进为线性（log(1/ε)）。
• 对于矩阵分解，该框架提供了尖锐的全局收敛速率和明确的迭代界。
• 该分析引入鞍点附近的几何表征（一个细长的“带”）以界定扰动后逃逸概率。
• 结果在最大步长 Ω(1/ℓ) 下成立，与一阶分析相当。