[Paper Review] On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Xiaoyu Li, Francesco Orabona|arXiv (Cornell University)|May 21, 2018

Stochastic Gradient Optimization TechniquesComputer Science107 citations

TL;DR

This paper analyzes a generalized AdaGrad-style adaptive stepsize for SGD, proving almost sure convergence to zero gradients in both non-convex and convex settings, and showing adaptive finite-time rates that interpolate between GD and SGD while adapting to gradient noise.

ABSTRACT

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in our theoretical understanding of these methods, especially in the non-convex setting. In this paper, we start closing this gap: we theoretically analyze in the convex and non-convex settings a generalized version of the AdaGrad stepsizes. We show sufficient conditions for these stepsizes to achieve almost sure asymptotic convergence of the gradients to zero, proving the first guarantee for generalized AdaGrad stepsizes in the non-convex setting. Moreover, we show that these stepsizes allow to automatically adapt to the level of noise of the stochastic gradients in both the convex and non-convex settings, interpolating between $O(1/T)$ and $O(1/\\sqrt{T})$, up to logarithmic terms.

Motivation & Objective

• Motivate and analyze adaptive stepsizes for SGD beyond convex/bounded-domain assumptions.
• Establish almost sure convergence of gradients to zero with generalized AdaGrad stepsizes in both convex and non-convex settings.
• Show that adaptive stepsizes automatically adapt to gradient-noise levels, interpolating between GD and SGD rates.

Proposed method

• Study two generalized AdaGrad-like stepsize rules: global stepsize eta_t = alpha / (beta + sum_{i=1}^{t-1} ||g(x_i, xi_i)||^2)^{1/2 + epsilon} and coordinate-wise eta_{t,j} = alpha / (beta + sum_{i=1}^{t-1} g(x_i, xi_i)_j^2)^{1/2 + epsilon}.
• Prove almost-sure convergence of SGD gradients to zero under Lipschitz smoothness, bounded-support noise, and the above stepsizes.
• Derive finite-time adaptive convergence rates in the convex setting showing interpolation between GD (when noise is small) and SGD (when noise is large).
• Provide non-convex convergence rates for the best iterate with adaptive stepsizes, showing adaptation to noise without knowing the noise level.

Experimental results

Research questions

• RQ1 Do generalized AdaGrad stepsizes ensure almost-sure convergence of gradients to zero in non-convex settings?
• RQ2 Can adaptive stepsizes adapt to gradient noise to yield finite-time rates that interpolate between GD and SGD in convex problems?
• RQ3 Do similar adaptive rates hold for non-convex settings, particularly for the best iterate rather than the last iterate?

Key findings

• SGD with generalized AdaGrad stepsizes converges almost surely to zero gradients in both non-convex and convex cases.
• In convex problems, the method adapts to noise levels and interpolates between GD and SGD rates up to polylog terms.
• In non-convex settings, the generalized AdaGrad stepsizes yield adaptive finite-time rates that improve with low noise, and extend to the best-iterate guarantee.
• The analysis demonstrates the first theoretical support for AdaGrad-like stepsizes offering advantages over plain SGD in non-convex optimization.