[论文解读] Katyusha Acceleration for Convex Finite-Sum Compositional Optimization

Yibo Xu, Yangyang Xu|arXiv (Cornell University)|Oct 24, 2019

Stochastic Gradient Optimization Techniques参考文献 33被引用 5

一句话总结

本文提出了一类用于凸有限和复合优化问题（COPs）的Katyusha加速算法，通过小批量采样与方差减少技术，实现了更优的收敛速率。对于强凸COPs，建立了复杂度依赖于$\kappa^{2.5}$的线性收敛，优于先前最优的$\kappa^3$；对于凸COPs，实现了$\varepsilon^{-2.5}$的复杂度，优于先前的$\varepsilon^{-3}$。

ABSTRACT

Structured problems arise in many applications. To solve these problems, it is important to leverage the structure information. This paper focuses on convex problems with a finite-sum compositional structure. Finite-sum problems appear as the sample average approximation of a stochastic optimization problem and also arise in machine learning with a huge amount of training data. One popularly used numerical approach for finite-sum problems is the stochastic gradient method (SGM). However, the additional compositional structure prohibits easy access to unbiased stochastic approximation of the gradient, so directly applying the SGM to a finite-sum compositional optimization problem (COP) is often inefficient. We design new algorithms for solving strongly-convex and also convex two-level finite-sum COPs. Our design incorporates the Katyusha acceleration technique and adopts the mini-batch sampling from both outer-level and inner-level finite-sum. We first analyze the algorithm for strongly-convex finite-sum COPs. Similar to a few existing works, we obtain linear convergence rate in terms of the expected objective error, and from the convergence rate result, we then establish complexity results of the algorithm to produce an $\varepsilon$-solution. Our complexity results have the same dependence on the number of component functions as existing works. However, due to the use of Katyusha acceleration, our results have better dependence on the condition number $κ$ and improve to $κ^{2.5}$ from the best-known $κ^3$. Finally, we analyze the algorithm for convex finite-sum COPs, which uses as a subroutine the algorithm for strongly-convex finite-sum COPs. Again, we obtain better complexity results than existing works in terms of the dependence on $\varepsilon$, improving to $\varepsilon^{-2.5}$ from the best-known $\varepsilon^{-3}$.

研究动机与目标

• 设计高效的一阶算法，用于具有两级有限和结构的凸有限和复合优化问题（COPs）。
• 利用有限和与复合结构，提升收敛速率，超越标准随机梯度方法。
• 在条件数$\kappa$与精度$\varepsilon$上的依赖关系上，优于现有方法。
• 将原本用于非复合问题的Katyusha加速方法，拓展至有限和复合设置。
• 建立强凸与凸情况下梯度与雅可比矩阵评估次数的复杂度界。

提出的方法

• 为强凸COPs提出一种新算法，结合Katyusha加速与来自外层与内层的双重小批量采样。
• 采用针对两级有限和结构定制的方差减少技术，以稳定并加速收敛。
• 引入一种新颖的基于动量的更新规则，受Katyusha启发，结合外推与递归梯度追踪。
• 对于凸COPs，将强凸算法作为子程序，结合自适应步长以实现更优收敛。
• 对$F_i$与$G_j$组件同时应用小批量采样，以降低方差并提升计算效率。
• 在光滑性与强凸性标准假设下，通过分析期望目标误差的衰减，推导理论复杂度界。

实验结果

研究问题

• RQ1 Katyusha加速能否有效适配具有两级结构的有限和复合优化问题？
• RQ2 对于强凸COPs，算法复杂度在条件数$\kappa$上的最优依赖关系为何？
• RQ3 能否将凸COPs的收敛速率提升至优于先前方法的$\varepsilon^{-3}$界限？
• RQ4 从两级同时进行小批量采样，如何影响算法的收敛性与复杂度？
• RQ5 达到$\varepsilon$-解的理论复杂度（以梯度与雅可比矩阵评估次数计）为何？

主要发现

• 对于强凸有限和COPs，该算法实现了$\kappa^{2.5}$复杂度依赖的线性收敛，优于已知最优的$\kappa^3$。
• 凸COPs的收敛速率为$\varepsilon^{-2.5}$，优于先前最优的$\varepsilon^{-3}$。
• 该算法在组件函数数量$n_1, n_2$上的依赖关系与现有方法一致，但在$\kappa$与$\varepsilon$上实现了改进。
• 数值结果表明，该方法在多次试验与多种参数设置下，均表现出稳定且优于VRSC-PG、C-SAGA、GC-SAGA与SCVRG的性能。
• 该方法在不牺牲有限和结构的前提下，实现了更优的复杂度，通过结合小批量采样与Katyusha风格动量。
• 理论分析证实，复杂度的提升源于复合设置中有效的方差减少与动量加速。