[论文解读] First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time

Yi Xu, Rong Jin+1 more|arXiv (Cornell University)|Nov 3, 2017

Sparse and Compressive Sensing Techniques被引用 58

一句话总结

本文提出 NEON，一种从 Hessian 提取负曲率的基于一阶随机过程的方法，能够几乎线性时间逃离鞍点并以高概率找到近似二阶驻点。

ABSTRACT

Two classes of methods have been proposed for escaping from saddle points with one using the second-order information carried by the Hessian and the other adding the noise into the first-order information. The existing analysis for algorithms using noise in the first-order information is quite involved and hides the essence of added noise, which hinder further improvements of these algorithms. In this paper, we present a novel perspective of noise-adding technique, i.e., adding the noise into the first-order information can help extract the negative curvature from the Hessian matrix, and provide a formal reasoning of this perspective by analyzing a simple first-order procedure. More importantly, the proposed procedure enables one to design purely first-order stochastic algorithms for escaping from non-degenerate saddle points with a much better time complexity (almost linear time in terms of the problem's dimensionality). In particular, we develop a {\\bf first-order stochastic algorithm} based on our new technique and an existing algorithm that only converges to a first-order stationary point to enjoy a time complexity of {$\\widetilde O(d/\\epsilon^{3.5})$ for finding a nearly second-order stationary point $\\bf{x}$ such that $\\|\ abla F(bf{x})\\|\\leq \\epsilon$ and $\ abla^2 F(bf{x})\\geq -\\sqrt{\\epsilon}I$ (in high probability), where $F(\\cdot)$ denotes the objective function and $d$ is the dimensionality of the problem. To the best of our knowledge, this is the best theoretical result of first-order algorithms for stochastic non-convex optimization, which is even competitive with if not better than existing stochastic algorithms hinging on the second-order information.

研究动机与目标

• 激发并解决随机非凸优化问题。
• 开发一阶过程，通过来自噪声的负曲率源（NEON）逃离非退化鞍点。
• 提供一个利用一阶信息实现二阶收敛保证的框架。
• 在问题维度上实现接近线性时间复杂度，以寻找近似二阶驻点。

提出的方法

• 引入 NEON：一种从噪声出发从 Hessian 提取负曲率的过程。
• 将 NEON 集成到一个通用的一阶随机算法框架中。
• 证明在寻找近似二阶驻点方面的二阶收敛保证。
• 推导时间复杂度结果，并展示几乎与问题维度线性相关。
• 将该框架与包含许多分量的有限和设置联系起来。

实验结果

研究问题

• RQ1 一阶随机方法是否能够通过利用来自噪声自然产生的负曲率来高效地逃离鞍点？
• RQ2 在随机非凸优化中，利用一阶信息找到近似二阶驻点的时间复杂度是多少？
• RQ3 如何将 NEON 集成到通用的 SGD 型算法中，以高概率保证二阶收敛？
• RQ4 整体算法在维度上的运行时间接近线性到何种程度？
• RQ5 所提出的方法是否适用于期望形式的问题和大规模有限和问题？

主要发现

• 提出 NEON，通过基于噪声的序列从 Hessian 提取负曲率。
• 开发一个框架，使用纯一阶随机方法实现二阶收敛保证。
• 证明最佳时间复杂度为 ~O(d/ε^{3.5})，以高概率找到一个点使 ∥∇F(x)∥ ≤ ε 且 ∇^2F(x) ≥ −√ε I。
• 证明在逃离鞍点方面，问题维度上的时间几乎呈线性。
• 一阶随机算法达到的近似二阶驻点与使用二阶信息的方法具有竞争力。