[论文解读] RSG: Beating SGD without Smoothness and/or Strong Convexity

Tianbao Yang, Qihang Lin|arXiv (Cornell University)|Dec 9, 2015

Sparse and Compressive Sensing Techniques参考文献 26被引用 5

一句话总结

本文提出了重启随机次梯度（RSG）方法，通过利用局部性质（如局部半强凸性和（Kurdyka-）Łojasiewicz性质），在非光滑和/或非强凸优化中实现了比原始SGD更优的迭代复杂度。在局部半强凸性下，RSG的复杂度为$O(\frac{1}{\epsilon}\log(\frac{1}{\epsilon}))$；在$\beta$次幂Łojasiewicz条件下，复杂度为$O(\frac{1}{\epsilon^{2\beta}}\log(\frac{1}{\epsilon}))$，优于SGD的$O(\frac{1}{\epsilon^2})$速率。

ABSTRACT

In this paper, we propose a novel {\bf R}estarted {\bf S}tochastic sub{\bf G}radient (RSG) method that can find an $\epsilon$-optimal solution for a broad class of non-smooth and/or non-strongly convex optimization problems with a lower iteration complexity than vanilla stochastic subgradient descent (SGD). In particular, for general non-smooth and non-strongly convex optimization, RSG reduces the dependence of SGD's iteration complexity on the distance to the optimal set of the initial solution to that of points on the $\epsilon$-level set. For a special family of non-smooth and non-strongly convex optimization problems whose epigraph is a polyhedron, we further show that RSG could converge linearly. In addition, RSG has an $O(\frac{1}{\epsilon}\log(\frac{1}{\epsilon}))$ iteration complexity for problems with a much weaker notion of strong convexity, namely locally semi-strongly convexity. For a family of (non-) smooth optimization problems that admit a local (Kurdyka-) \L ojasiewicz property with a power constant of $\beta\in(0,1)$, RSG has an $O(\frac{1}{\epsilon^{2\beta}}\log(\frac{1}{\epsilon}))$ iteration complexity, which is better that that of SGD for such optimization problems whose iteration complexity is $O(\frac{1}{\epsilon^2})$. The novelty of our analysis lies at exploiting the lower bound of the subgradient or gradient of the objective function at the $\epsilon$-level set. It is this novelty that allows us to explore the local properties of functions (e.g., local semi-strong convexity, local (Kurdyka-) \L ojasiewicz property, more generally local error bounds) to develop improved convergence of RSG.

研究动机与目标

• 解决在非光滑和非强凸设置下随机次梯度下降（SGD）的高迭代复杂度问题。
• 通过利用$\epsilon$-水平集的性质，减少对初始距离到最优集的依赖。
• 为具有较弱凸性或误差界结构的问题，实现比SGD更快的收敛速率。
• 通过利用$\epsilon$-水平集上次梯度的下界，建立改进的迭代复杂度界。

提出的方法

• 提出重启随机次梯度（RSG）方法，通过周期性重启加速收敛。
• 采用基于$\epsilon$-水平集内点处次梯度下界的新型分析技术。
• 当当前迭代点进入$\epsilon$-水平集时触发重启，利用目标函数的局部正则性。
• 利用局部误差界，包括幂常数$\beta \in (0,1)$的（Kurdyka-）Łojasiewicz性质，推导出改进的收敛速率。
• 推导出依赖于目标函数局部几何结构而非全局假设的迭代复杂度界。
• 提出一个框架，将收敛保证推广至具有弱凸性结构的非光滑和非强凸问题。

实验结果

研究问题

• RQ1 能否在非光滑和非强凸问题中，使随机次梯度方法的迭代复杂度优于SGD？
• RQ2 RSG方法如何利用局部性质（如局部半强凸性）来提升收敛性？
• RQ3 在幂常数$\beta \in (0,1)$的（Kurdyka-）Łojasiewicz条件下，RSG的迭代复杂度是多少？
• RQ4 RSG能否在上境图（epigraph）为多面体的一类问题中实现线性收敛？
• RQ5 RSG对初始距离到最优集的依赖性与原始SGD相比如何？

主要发现

• RSG通过将迭代复杂度与$\epsilon$-水平集上点到最优集的距离关联，降低了对初始距离的依赖。
• 对于上境图为多面体的问题，RSG在非光滑且不具强凸性的情况下仍能实现线性收敛。
• 在局部半强凸条件下，RSG实现$O(\frac{1}{\epsilon}\log(\frac{1}{\epsilon}))$的迭代复杂度，优于SGD的$O(\frac{1}{\epsilon^2})$速率。
• 对于满足$\beta$次幂（Kurdyka-）Łojasiewicz条件（$\beta \in (0,1)$）的问题，RSG实现$O(\frac{1}{\epsilon^{2\beta}}\log(\frac{1}{\epsilon}))$的迭代复杂度，当$\beta < 1/2$时，快于SGD的$O(\frac{1}{\epsilon^2})$。
• 关键创新在于利用$\epsilon$-水平集上次梯度的下界，以捕捉局部函数几何结构并推导出更紧的收敛界。