[论文解读] How Good is SGD with Random Shuffling?

Itay Safran, Ohad Shamir|arXiv (Cornell University)|Jul 31, 2019

Stochastic Gradient Optimization Techniques参考文献 15被引用 5

一句话总结

本文首次为有限和、光滑且强凸问题中的随机梯度下降（SGD）在随机重排下的优化误差提供了紧致的下界。证明了重复随机重排相较于单次重排（$\Omega(1/(nk^2))$）实现了更快的收敛速率 $\Omega(1/(nk)^2 + 1/(nk^3))$，在固定步长下确立了两种策略之间的固有性能差距。

ABSTRACT

We study the performance of stochastic gradient descent (SGD) on smooth and strongly-convex finite-sum optimization problems. In contrast to the majority of existing theoretical works, which assume that individual functions are sampled with replacement, we focus here on popular but poorly-understood heuristics, which involve going over random permutations of the individual functions. This setting has been investigated in several recent works, but the optimal error rates remain unclear. In this paper, we provide lower bounds on the expected optimization error with these heuristics (using SGD with any constant step size), which elucidate their advantages and disadvantages. In particular, we prove that after $k$ passes over $n$ individual functions, if the functions are re-shuffled after every pass, the best possible optimization error for SGD is at least $Ω\left(1/(nk)^2+1/nk^3 ight)$, which partially corresponds to recently derived upper bounds. Moreover, if the functions are only shuffled once, then the lower bound increases to $Ω(1/nk^2)$. Since there are strictly smaller upper bounds for repeated reshuffling, this proves an inherent performance gap between SGD with single shuffling and repeated shuffling. As a more minor contribution, we also provide a non-asymptotic $Ω(1/k^2)$ lower bound (independent of $n$) for the incremental gradient method, when no random shuffling takes place. Finally, we provide an indication that our lower bounds are tight, by proving matching upper bounds for univariate quadratic functions.

研究动机与目标

• 理解SGD在有限和优化中采用随机重排的理论性能极限。
• 解决重复随机重排是否在理论上优于单次重排或固定顺序增量方法的开放问题。
• 为不同重排策略下常数步长SGD的优化误差建立紧致下界。
• 通过证明单变量二次函数下匹配的下界，展示现有上界的最佳性。

提出的方法

• 推导重复随机重排、单次重排及增量梯度方法下SGD期望优化误差的下界。
• 采用单变量二次函数模型，构建与已知上界匹配的紧致下界。
• 应用浓度不等式和AM-GM不等式，以界定误差递推中收缩因子的乘积。
• 通过递归期望界分析 $k$ 次遍历中的误差传播，并利用重排迭代之间的独立性。
• 为无随机重排的增量梯度方法建立非渐近的 $\Omega(1/k^2)$ 下界。
• 采用步长 $\eta = \frac{\log(n^{0.5}k)}{\lambda n k}$，以在分析中平衡收敛项与误差项。

实验结果

研究问题

• RQ1 在光滑、强凸的有限和问题中，SGD采用重复随机重排的最优可实现误差率是多少？
• RQ2 重复随机重排是否在理论上优于单次重排或固定顺序增量方法？
• RQ3 在不同重排方案下，常数步长SGD的优化误差是否存在根本性下界？
• RQ4 现有重复重排的上界是否紧致，还是可以进一步改进？
• RQ5 缺乏随机重排如何影响增量梯度方法的收敛速率？

主要发现

• SGD采用重复随机重排的期望优化误差至少为 $\Omega(1/(nk)^2 + 1/(nk^3))$，与已知上界匹配。
• 对于单次重排，下界为 $\Omega(1/(nk^2))$，严格劣于重复重排情形。
• 这确立了重复重排与单次重排之间的固有性能差距，证明重复重排在理论上更优。
• 对于增量梯度方法（无随机重排），建立了与 $n$ 无关的非渐近 $\Omega(1/k^2)$ 下界。
• 对于单变量二次函数，下界与上界匹配，表明分析的紧致性。
• 分析确认，在重复重排下，最优可能的误差率不会优于 $\tilde{\mathcal{O}}(1/(nk)^2 + 1/(nk^3))$，且该界是紧致的。