[论文解读] Linear Convergence of Variance-Reduced Projected Stochastic Gradient without Strong Convexity.

Pinghua Gong, Jieping Ye|arXiv (Cornell University)|Jun 4, 2014

Stochastic Gradient Optimization Techniques被引用 12

一句话总结

本文提出了方差缩减投影随机梯度（VRPSG）方法，并针对非强凸优化问题建立了线性收敛性，此类问题在机器学习中极为常见。通过证明一种新型半强凸（SSC）不等式，作者表明VRPSG与Prox-SVRG在无需强凸性假设下仍能实现线性收敛，这是首次在约束和正则化设置下获得此类结果。

ABSTRACT

Stochastic gradient algorithms estimate the gradient based on only one or a few samples and enjoy low computational cost per iteration. They have been widely used in large-scale optimization problems. However, stochastic gradient algorithms are usually slow to converge and achieve sub-linear convergence rates, due to the inherent variance in the gradient computation. To accelerate the convergence, some variance-reduced stochastic gradient algorithms, e.g., proximal stochastic variance-reduced gradient (Prox-SVRG) algorithm, have recently been proposed to solve strongly convex problems. Under the strongly convex condition, these variance-reduced stochastic gradient algorithms achieve a linear convergence rate. However, many machine learning problems are convex but not strongly convex. In this paper, we introduce Prox-SVRG and its projected variant called Variance-Reduced Projected Stochastic Gradient (VRPSG) to solve a class of non-strongly convex optimization problems widely used in machine learning. As the main technical contribution of this paper, we show that both VRPSG and Prox-SVRG achieve a linear convergence rate without strong convexity. A key ingredient in our proof is a Semi-Strongly Convex (SSC) inequality which is the first to be rigorously proved for a class of non-strongly convex problems in both constrained and regularized settings. Moreover, the SSC inequality is independent of algorithms and may be applied to analyze other stochastic gradient algorithms besides VRPSG and Prox-SVRG, which may be of independent interest. To the best of our knowledge, this is the first work that establishes the linear convergence rate for the variance-reduced stochastic gradient algorithms on solving both constrained and regularized problems without strong convexity.

研究动机与目标

• 为解决现有方差缩减随机梯度方法在缺乏强凸性时无法实现线性收敛的局限性。
• 将线性收敛性保证扩展至机器学习应用中普遍存在的非强凸问题。
• 构建一个理论框架，支持在无强凸性假设下，对约束和正则化优化问题实现线性收敛。
• 提出并严格证明半强凸（SSC）不等式，作为非强凸问题的关键分析工具。

提出的方法

• 提出Prox-SVRG的投影变体，命名为方差缩减投影随机梯度（VRPSG），用于求解约束优化问题。
• 引入一种新的理论条件——半强凸（SSC）不等式，该不等式适用于一类非强凸问题。
• 证明SSC不等式可使VRPSG与Prox-SVRG在无强凸性条件下实现线性收敛。
• 将SSC不等式作为核心分析工具，用于独立于算法设计地界定向每轮迭代的进展。
• 将SSC条件应用于正则化和约束优化设置，证明其广泛适用性。
• 通过证明在SSC条件下，期望次优性呈指数衰减，从而确立收敛速率保持线性。

实验结果

研究问题

• RQ1 方差缩减随机梯度方法能否在非强凸问题中实现线性收敛？
• RQ2 在缺乏强凸性时，何种理论条件可实现线性收敛？
• RQ3 半强凸（SSC）不等式是否为非强凸问题的有效且可推广的条件？
• RQ4 SSC不等式是否可用于分析VRPSG与Prox-SVRG之外的其他随机梯度算法？
• RQ5 所提出的VRPSG方法在无强凸性假设下，是否在约束和正则化设置中均保持线性收敛？

主要发现

• VRPSG在非强凸问题中实现了线性收敛，这是此前该类问题中尚未被证明的结果。
• 所提出的半强凸（SSC）不等式经过严格证明，适用于一大类非强凸问题。
• SSC不等式与优化算法无关，可应用于分析其他随机梯度方法。
• 在无强凸性假设下，已为约束和正则化优化问题确立线性收敛速率。
• 本工作首次展示了方差缩减随机梯度算法在非强凸设置下的线性收敛性。
• 基于SSC的理论框架为大规模机器学习问题中的收敛性分析提供了新路径。