[Paper Review] AdaGrad stepsizes: Sharp convergence over nonconvex landscapes

Rachel Ward, Xiaoxia Wu+1 more|arXiv (Cornell University)|Jun 5, 2018

Advanced Optimization Algorithms ResearchMathematics136 citations

TL;DR

AdaGrad-Norm converges to stationary points in smooth nonconvex optimization, achieving an O(log(N)/sqrt(N)) rate in the stochastic setting and an O(1/N) rate in the deterministic setting, with robustness to hyperparameters.

ABSTRACT

Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule. Yet, the theoretical guarantees to date for AdaGrad are for online and convex optimization. We bridge this gap by providing theoretical guarantees for the convergence of AdaGrad for smooth, nonconvex functions. We show that the norm version of AdaGrad (AdaGrad-Norm) converges to a stationary point at the $\mathcal{O}(\log(N)/\sqrt{N})$ rate in the stochastic setting, and at the optimal $\mathcal{O}(1/N)$ rate in the batch (non-stochastic) setting -- in this sense, our convergence guarantees are 'sharp'. In particular, the convergence of AdaGrad-Norm is robust to the choice of all hyper-parameters of the algorithm, in contrast to stochastic gradient descent whose convergence depends crucially on tuning the step-size to the (generally unknown) Lipschitz smoothness constant and level of stochastic noise on the gradient. Extensive numerical experiments are provided to corroborate our theory; moreover, the experiments suggest that the robustness of AdaGrad-Norm extends to state-of-the-art models in deep learning, without sacrificing generalization.

Motivation & Objective

• Motivate robust optimization without tuning exact Lipschitz constants or noise levels.
• Provide theoretical convergence guarantees for AdaGrad-Norm in smooth, nonconvex settings.
• Derive stochastic and deterministic convergence rates and clarify hyperparameter impact.
• Offer practical guidance for setting hyperparameters when L and noise are unknown.

Proposed method

• Define the AdaGrad-Norm update: x_{j+1} = x_j - (η / b_{j+1}) G_j with b_{j+1}^{2} = b_j^{2} + ||G_j||^{2}.
• Assume G_j is an unbiased gradient estimator with bounded variance and gradient norms, and that ||∇F(x)|| ≤ γ.
• Prove convergence results (Theorems 2.1 and 2.2) for stochastic and deterministic settings respectively.
• Utilize the Descent Lemma and auxiliary bounds to handle the correlated randomness between b_j and G_j.
• Provide rate statements and compare to SGD with fixed step sizes, highlighting robustness to hyperparameters.
• Offer practical parameter choices when F* is known (η = F(x0) − F*) and b0 small.

Experimental results

Research questions

• RQ1 Does AdaGrad-Norm converge to stationary points for smooth nonconvex F under stochastic gradients?
• RQ2 What are the convergence rates of AdaGrad-Norm in stochastic and deterministic settings, and how do hyperparameters affect them?
• RQ3 Is AdaGrad-Norm robust to arbitrary positive choices of η and b0 without knowing the Lipschitz constant L or noise σ?
• RQ4 How do the constants in the convergence rates depend on initial conditions and hyperparameters?

Key findings

• In the stochastic setting, AdaGrad-Norm converges to an ε-approximate stationary point at rate O(log(N)/sqrt(N)).
• In the deterministic setting, AdaGrad-Norm achieves the optimal O(1/N) rate.
• The convergence holds for any η>0 and b0>0, showing robustness to hyperparameter choices.
• Convergence constants explicitly depend on b0 and η, and guidance is provided for practical parameter settings.
• Compared to SGD with fixed stepsize, AdaGrad-Norm delivers robust convergence without requiring prior knowledge of the smoothness L or noise σ.