[Paper Review] On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization

Dongruo Zhou, Jinghui Chen+3 more|arXiv (Cornell University)|Aug 16, 2018

Sparse and Compressive Sensing Techniques41 references82 citations

TL;DR

The paper provides fine-grained convergence analyses for adaptive gradient methods (AMSGrad, RMSProp, AdaGrad) in stochastic nonconvex optimization, giving expectation and high-probability guarantees and showing advantages over SGD under gradient sparsity.

ABSTRACT

Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSProp and AdaGrad. For smooth nonconvex functions, we prove that adaptive gradient methods in expectation converge to a first-order stationary point. Our convergence rate is better than existing results for adaptive gradient methods in terms of dimension. In addition, we also prove high probability bounds on the convergence rates of AMSGrad, RMSProp as well as AdaGrad, which have not been established before. Our analyses shed light on better understanding the mechanism behind adaptive gradient methods in optimizing nonconvex objectives.

Motivation & Objective

• Motivate the study of convergence guarantees for adaptive gradient methods in nonconvex stochastic optimization, addressing gaps between practice and theory.
• Develop a unified, fine-grained convergence analysis for AMSGrad, RMSProp, and AdaGrad under smooth nonconvex objectives.
• Establish both expectation-based and high-probability convergence bounds to first-order stationary points.
• Show improved iteration complexity and dimension dependence, particularly under gradient sparsity, compared to SGD.

Proposed method

• Analyze a general class of adaptive gradient methods (AMSGrad, RMSProp, AdaGrad) in stochastic nonconvex optimization with L-smooth objectives.
• Work under bounded-infinity stochastic gradient assumption (G_infty) and L-smoothness; introduce an auxiliary sequence z_t to handle stochastic momentum.
• Derive a convergence rate to stationary points: E[||∇f(x_t)||^2] ≤ M1/(Tα) + Md/T + αMd/(T^{1/2−s}) under growth condition ||g_{1:T,i}||_2 ≤ G_infty T^s.
• Provide high-probability convergence bounds by assuming sub-Gaussian stochastic gradients, yielding analogous rate forms with probabilistic guarantees.
• Compare with existing results, showing improved d-dependence and, in sparse gradient regimes (s<1/2), strictly better rates than SGD.

Experimental results

Research questions

• RQ1 Do AMSGrad, RMSProp, and AdaGrad converge to first-order stationary points in stochastic nonconvex optimization?
• RQ2 What are the exact convergence rates for these adaptive methods in terms of dimension d, iterations T, and gradient sparsity (captured by s)?
• RQ3 Can high-probability convergence guarantees be established for these adaptive methods in the nonconvex setting?
• RQ4 How do the derived rates compare to SGD, particularly under sparse gradients?
• RQ5 What role do algorithmic corrections (e.g., AMSGrad's non-increasing learning rates) play in ensuring convergence?

Key findings

• Convergence to stationary points is proven for AMSGrad, RMSProp, and AdaGrad in expectation for smooth nonconvex objectives.
• AMSGrad achieves a rate of O(d^{1/2}/T^{3/4−s/2} + d/T) under a growth bound on cumulative gradients, improving over SGD when gradients are sparse (s < 1/2).
• In the worst case (s = 1/2), the rate is O(√d/√T + d/T), with better dimension dependence than prior bounds.
• High-probability convergence bounds are established for AMSGrad, RMSProp, and AdaGrad, addressing single-run performance guarantees.
• AdaGrad, RMSProp, and AMSGrad share the same rate form under the stated conditions, with explicit constants provided for each method.
• The analysis introduces an auxiliary sequence z_t to handle stochastic momentum and proves auxiliary-boundedness lemmas to enable the rate derivations.