[论文解读] First-order methods almost always avoid saddle points: the case of vanishing step-sizes

Ioannis Panageas, Georgios Piliouras+1 more|arXiv (Cornell University)|Jun 18, 2019

Gaussian Processes and Bayesian Inference参考文献 21被引用 4

一句话总结

本文证明，即使在使用阶为 Ω(1/k) 的趋近于零的步长时，一阶优化方法——包括梯度下降、镜像下降、邻近点法和流形下降——几乎必然避开严格鞍点。通过为时变非齐次动力系统建立一个新颖的稳定流形定理，作者们将先前针对恒定步长的结果推广至自适应和递减步长的情形，从而解决了非凸优化领域长期悬而未决的开放问题。

ABSTRACT

In a series of papers \cite{LSJR16, PP17, LPP}, it was established that some of the most commonly used first order methods almost surely (under random initializations) and with step-size being small enough, avoid strict saddle points, as long as the objective function $f$ is $C^2$ and has Lipschitz gradient. The key observation was that first order methods can be studied from a dynamical systems perspective, in which instantiations of Center-Stable manifold theorem allow for a global analysis. The results of the aforementioned papers were limited to the case where the step-size $α$ is constant, i.e., does not depend on time (and bounded from the inverse of the Lipschitz constant of the gradient of $f$). It remains an open question whether or not the results still hold when the step-size is time dependent and vanishes with time. In this paper, we resolve this question on the affirmative for gradient descent, mirror descent, manifold descent and proximal point. The main technical challenge is that the induced (from each first order method) dynamical system is time non-homogeneous and the stable manifold theorem is not applicable in its classic form. By exploiting the dynamical systems structure of the aforementioned first order methods, we are able to prove a stable manifold theorem that is applicable to time non-homogeneous dynamical systems and generalize the results in \cite{LPP} for vanishing step-sizes.

研究动机与目标

• 解决一阶方法在使用趋近于零的步长时是否避开严格鞍点的开放问题。
• 将稳定流形理论的应用范围从时齐次系统扩展至由一阶方法诱导的时非齐次动力系统。
• 证明在随机初始化和趋近于零的步长条件下，梯度下降、镜像下降、邻近点法和流形下降几乎必然避开严格鞍点。
• 为实践中常用的自适应和递减步长调度建立理论基础。

提出的方法

• 为时变非齐次动力系统开发一个新稳定流形定理，推广经典稳定流形定理。
• 分析在时变步长下，梯度下降、镜像下降、邻近点法和流形下降等一阶方法的动力学行为。
• 通过黎曼度量变换对临界点处的雅可比矩阵进行对角化，并分析其局部线性化行为。
• 证明在优化方法诱导的黎曼度量下，严格鞍点的稳定集测度为零，从而表明几乎必然避开。
• 验证更新规则泰勒展开中的余项满足新定理所需的正则性条件。
• 应用推广后的稳定流形定理，表明从随机初始化出发的轨迹几乎必然避开严格鞍点。

实验结果

研究问题

• RQ1 当使用趋近于零的步长时，一阶优化方法是否避开严格鞍点？
• RQ2 经典稳定流形定理能否推广至由自适应或递减步长引起的时非齐次动力系统？
• RQ3 在趋近于零的步长条件下，镜像下降、邻近点法和流形下降是否仍保持对严格鞍点的几乎必然避开？
• RQ4 何种步长序列条件可确保在非凸优化中对严格鞍点的几乎必然避开？

主要发现

• 在随机初始化下，步长为 αk = Ω(1/k) 的梯度下降几乎必然避开严格鞍点。
• 在相同步长条件下，镜像下降、邻近点法和流形下降也具有相同的避开性质。
• 为时变非齐次动力系统证明了一个新的稳定流形定理，使得能够对非自治一阶方法进行全局分析。
• 在优化方法诱导的黎曼度量下，严格鞍点的稳定集测度为零，从而确保几乎必然避开。
• 通过在函数 x² − y² 上的模拟验证了理论结果，表明趋近于零的步长可避开原点（一个严格鞍点），而极慢衰减（1/k⁴）则失败。
• 分析表明，步长的选择显著影响收敛行为，较慢的衰减可能导致收敛至非临界点。