[论文解读] Unified Optimal Analysis of the (Stochastic) Gradient Method

Sebastian U. Stich|arXiv (Cornell University)|Jul 9, 2019

Stochastic Gradient Optimization Techniques参考文献 21被引用 55

一句话总结

这篇论文在 (L, sigma)-光滑性条件下，对基于 mu-凸性的函数上的 SGD 提出了一种简单、统一的收敛性分析，获得指数收敛速率加上一个随机项，并在插值情形下恢复了 GD/SGD 的已知速率。

ABSTRACT

In this note we give a simple proof for the convergence of stochastic gradient (SGD) methods on $μ$-convex functions under a (milder than standard) $L$-smoothness assumption. We show that for carefully chosen stepsizes SGD converges after $T$ iterations as $O\left( LR^2 \exp \bigl[-\fracμ{4L}T\bigr] + \frac{σ^2}{μT} ight)$ where $σ^2$ measures the variance in the stochastic noise. For deterministic gradient descent (GD) and SGD in the interpolation setting we have $σ^2 =0$ and we recover the exponential convergence rate. The bound matches with the best known iteration complexity of GD and SGD, up to constants.

研究动机与目标

• Motivate a milder yet practical (L, sigma)-smoothness assumption for SGD.
• Provide a simple, unified convergence proof that covers both deterministic GD and SGD.
• Show optimal or near-optimal rates for function suboptimality and last-iterate distance to optimality under mu-convexity.
• Demonstrate that the analysis recovers exponential convergence in the deterministic/interpolation setting.
• Offer insights into averaging schemes that achieve fast decay of the stochastic error term.

提出的方法

• Analyze SGD with unbiased gradient oracle under (L, sigma)-smoothness and mu-convexity.
• Derive a recursion for the expected squared distance to the optimum and the suboptimality f(x_t) - f*, using a stepsize constraint gamma_t <= 1/(2L).
• Obtain a bound showing E[f(x̄_T) - f* + mu E||x_{T+1}-x*||^2] = O( L R^2 exp(-mu T/(4L)) + sigma^2/(mu T) ).
• Introduce a two-phase averaging scheme to balance optimization and variance reduction.
• Show how constant and decreasing stepsizes yield complementary convergence guarantees.
• Connect the recursion to established results to recover known rates for GD and SGD, including interpolation.

实验结果

研究问题

• RQ1 What convergence rates can be guaranteed for SGD under mu-convexity and an (L, sigma)-smoothness condition?
• RQ2 Can a simple, unified proof recover both the exponential rates (deterministic/interpolation) and stochastic rates for function values and last-iterate distance?
• RQ3 How should stepsizes and averaging be chosen to optimize the trade-off between optimization error and stochastic variance?
• RQ4 Do the results extend to common SGD settings, including interpolation and standard stochastic gradients, without bounded gradient assumptions?

主要发现

• For SGD with appropriately chosen stepsizes, the expected function suboptimality plus a mu-weighted last-iterate distance converges as O(L R^2 exp(-mu T/(4L)) + sigma^2/(mu T)).
• In the interpolation setting (sigma^2 = 0), the bound yields exponential convergence for function values and last-iterate distance, up to constants.
• The analysis recovers the best-known iteration complexity of GD and SGD up to constants, under the stated assumptions.
• A two-phase averaging scheme (initial non-averaged phase followed by suffix averaging) achieves optimal rates for the stochastic term without sacrificing the optimization term.
• The framework unifies analyses of gradient descent and SGD for smooth functions and does not rely on bounded-gradient assumptions.