[论文解读] Stability and Convergence Trade-off of Iterative Optimization Algorithms

Yuansi Chen, Chi Jin+1 more|arXiv (Cornell University)|Apr 4, 2018

Stochastic Gradient Optimization Techniques参考文献 2被引用 42

一句话总结

本论文在学习中的迭代优化中建立了收敛速率与算法稳定性之间的基本权衡，表明优化误差与稳定性之和在下界被 minimax 统计误差所界定，并推导出在凸和强凸设定下对 GD、SGD、NAG、HB 的下界。

ABSTRACT

The overall performance or expected excess risk of an iterative machine learning algorithm can be decomposed into training error and generalization error. While the former is controlled by its convergence analysis, the latter can be tightly handled by algorithmic stability. The machine learning community has a rich history investigating convergence and stability separately. However, the question about the trade-off between these two quantities remains open. In this paper, we show that for any iterative algorithm at any iteration, the overall performance is lower bounded by the minimax statistical error over an appropriately chosen loss function class. This implies an important trade-off between convergence and stability of the algorithm -- a faster converging algorithm has to be less stable, and vice versa. As a direct consequence of this fundamental tradeoff, new convergence lower bounds can be derived for classes of algorithms constrained with different stability bounds. In particular, when the loss function is convex (or strongly convex) and smooth, we discuss the stability upper bounds of gradient descent (GD) and stochastic gradient descent and their variants with decreasing step sizes. For Nesterov's accelerated gradient descent (NAG) and heavy ball method (HB), we provide stability upper bounds for the quadratic loss function. Applying existing stability upper bounds for the gradient methods in our trade-off framework, we obtain lower bounds matching the well-established convergence upper bounds up to constants for these algorithms and conjecture similar lower bounds for NAG and HB. Finally, we numerically demonstrate the tightness of our stability bounds in terms of exponents in the rate and also illustrate via a simulated logistic regression problem that our stability bounds reflect the generalization errors better than the simple uniform convergence bounds for GD and NAG.

研究动机与目标

• 激发在迭代学习算法中平衡优化收敛性与泛化性的必要性。
• 引入一个框架，以极小极大统计误差下界化优化误差与算法稳定性之和。
• 在凸与强凸损失设定下，推导常见一阶方法的稳定性上界及相应的收敛下界。
• 提供理论见解与数值示例，展示稳定性-收敛性权衡的实际意义。

提出的方法

• 将期望超额风险分解为泛化误差和优化误差，以研究权衡。
• 利用统一算法稳定性（Bousquet and Elisseeff, 2002）来界定泛化误差。
• 建立两类损失函数（凸光滑和强凸光滑），并证明将稳定性与收敛性联系起来的下界（定理7和定理9）。
• 在凸光滑设定下推导 GD、SGD、NAG、HB 的稳定性界（定理10-12），并给出将扩展到一般凸光滑损失的猜想。
• 应用 Le Cam 型极小极大论证将稳定性-收敛性权衡转化为具体的收敛下界。
• 进行数值仿真以验证速率指数并相对于统一收敛界展示泛化行为。

实验结果

研究问题

• RQ1 是否存在一个基本极限，将迭代优化算法的收敛速率与其在学习中的稳定性联系起来？
• RQ2 统一稳定性与优化误差如何共同界定凸与强凸光滑损失类中的期望超额风险？
• RQ3 基于稳定性的下界是否能再现 GD、SGD、NAG、HB 的已知收敛速率，以及对更快方法的影响？
• RQ4 在初始迭代阶段，稳定性考量是否能比经典的统一收敛界更准确地反映泛化误差？

主要发现

• 存在一个基本的权衡：优化误差与稳定性的和至少等于所选损失类上的极小极大统计误差。
• 对于凸光滑损失，极小极大速率阶为 1/√n；对于强凸光滑损失，为 1/n。
• 梯度下降和 SGD 的收敛下界在稳定性受限框架内与已知上界相符，常数项之外。
• Nesterov 加速梯度（NAG）和重球法（HB）的稳定性界表明它们在实现更快收敛的同时，不能像 GD 那样稳定，符合权衡。
• 该框架在不同稳定性界下为算法提供了新的收敛下界，数值仿真证实了这些速率以及相比于初始迭代的简单统一界更能反映泛化误差。
• 经验性示例（逻辑回归）表明稳定性界与泛化误差行为的对齐程度比简单的统一收敛界更高。