[Paper Review] The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares

Rong Ge, Sham M. Kakade+2 more|arXiv (Cornell University)|Apr 29, 2019

Stochastic Gradient Optimization Techniques61 references66 citations

TL;DR

The paper shows that SGD’s final iterate with polynomially decaying stepsizes is suboptimal for streaming least squares, and introduces a step decay (geometric) schedule that nearly attains minimax rates, with near-optimality up to log factors in known-horizon settings.

ABSTRACT

Minimax optimal convergence rates for classes of stochastic convex optimization problems are well characterized, where the majority of results utilize iterate averaged stochastic gradient descent (SGD) with polynomially decaying step sizes. In contrast, SGD's final iterate behavior has received much less attention despite their widespread use in practice. Motivated by this observation, this work provides a detailed study of the following question: what rate is achievable using the final iterate of SGD for the streaming least squares regression problem with and without strong convexity? First, this work shows that even if the time horizon T (i.e. the number of iterations SGD is run for) is known in advance, SGD's final iterate behavior with any polynomially decaying learning rate scheme is highly sub-optimal compared to the minimax rate (by a condition number factor in the strongly convex case and a factor of $\sqrt{T}$ in the non-strongly convex case). In contrast, this paper shows that Step Decay schedules, which cut the learning rate by a constant factor every constant number of epochs (i.e., the learning rate decays geometrically) offers significant improvements over any polynomially decaying step sizes. In particular, the final iterate behavior with a step decay schedule is off the minimax rate by only $log$ factors (in the condition number for strongly convex case, and in T for the non-strongly convex case). Finally, in stark contrast to the known horizon case, this paper shows that the anytime (i.e. the limiting) behavior of SGD's final iterate is poor (in that it queries iterates with highly sub-optimal function value infinitely often, i.e. in a limsup sense) irrespective of the stepsizes employed. These results demonstrate the subtlety in establishing optimal learning rate schemes (for the final iterate) for stochastic gradient procedures in fixed time horizon settings.

Motivation & Objective

• Characterize the final-iterate behavior of SGD on streaming least squares with/without strong convexity.
• Show suboptimality of polynomially decaying stepsizes for the final iterate.
• Propose and analyze a geometrically decaying step-decay schedule that approaches minimax rates.
• Contrast known-horizon results with anytime (limiting) behavior of SGD.
• Provide empirical validation on both synthetic least squares and a CIFAR-10 residual network.
• Discuss practical implications for hyperparameter tuning under horizon knowledge.

Proposed method

• Formalize SGD with stochastic gradient oracle under least squares, with assumptions on noise and fourth-moment covariates.
• Define step sizes: polynomial decay eta_t ~ a/(b+t^alpha) and step-decay (geometric) schedules (Algorithm 1).
• Derive lower bounds showing suboptimality of polynomial decay for final iterate in strongly and non-strongly convex cases.
• Prove upper bounds showing step-decay achieves near minimax rates, with excess risk bounds differing by only a log(T) factor.
• Present experiments comparing decay schemes on CIFAR-10 with a ResNet-44 and discuss suffix averaging effects.

Experimental results

Research questions

• RQ1 Can SGD’s final iterate match minimax rates for streaming least squares under fixed horizon T?
• RQ2 Do polynomially decaying stepsizes yield suboptimal final-iterate performance compared to step-decay schedules?
• RQ3 How close to minimax rates are achieved by step-decay schedules in strongly and non-strongly convex least squares?
• RQ4 What is the difference between known-horizon and anytime behavior of SGD’s final iterate?
• RQ5 Do empirical results on real-world networks support the theoretical benefits of step-decay schedules?

Key findings

• Polynomially decaying stepsizes yield suboptimal final-iterate rates, with gaps scaling by the condition number (strongly convex) or by sqrt(T)/log T (non-strongly convex).
• Step decay schedules achieve near minimax rates, with final-iterate excess risk off by only a log(T) factor in both strongly and non-strongly convex least squares under known horizon.
• In the strongly convex case, lower bounds show any final iterate with polynomial decay incurs a κ factor suboptimality; in the smooth case, a √T/log T gap is shown.
• The step-decay scheme requires only the initial learning rate and end time T for implementation; a refinement can reduce log factors to log(kappa) in the strongly convex case.
• Anytime (limiting) behavior of SGD’s final iterate remains poor regardless of stepsize scheme, with limsup suboptimality bounded away from minimax rates.
• Empirical results on CIFAR-10 with a ResNet-44 show continuous step-decay (exponential) often outperforms polynomial decays; suffix averaging can harm generalization in non-convex settings.