[Paper Review] A Simple Convergence Proof of Adam and Adagrad

Alexandre Défossez, Léon Bottou+2 more|arXiv (Cornell University)|Mar 5, 2020

Stochastic Gradient Optimization Techniques18 references32 citations

TL;DR

The paper provides a simple, unified convergence proof for Adagrad and Adam (with/without momentum) on smooth, possibly non-convex objectives, giving explicit gradient-norm bounds and improved dependence on momentum. It shows Adam can match Adagrad’s rate under proper parameters, and explains why default Adam may not converge.

ABSTRACT

We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients. We show that in expectation, the squared norm of the objective gradient averaged over the trajectory has an upper-bound which is explicit in the constants of the problem, parameters of the optimizer, the dimension $d$, and the total number of iterations $N$. This bound can be made arbitrarily small, and with the right hyper-parameters, Adam can be shown to converge with the same rate of convergence $O(d\ln(N)/\sqrt{N})$. When used with the default parameters, Adam doesn't converge, however, and just like constant step-size SGD, it moves away from the initialization point faster than Adagrad, which might explain its practical success. Finally, we obtain the tightest dependency on the heavy ball momentum decay rate $β_1$ among all previous convergence bounds for non-convex Adam and Adagrad, improving from $O((1-β_1)^{-3})$ to $O((1-β_1)^{-1})$.

Motivation & Objective

• Motivate and prove convergence guarantees for adaptive methods Adagrad and Adam (with and without momentum) on smooth, possibly non-convex objectives.
• Provide explicit upper bounds on the expected squared gradient norm along the optimization trajectory.
• Clarify how hyperparameters (learning rate, momentum, and beta parameters) affect convergence and rate.
• Compare Adagrad and Adam under a common analytical framework and discuss practical implications of default parameters.

Proposed method

• Use a unified stochastic optimization setup with per-coordinate adaptive steps and exponential moving averages of squared gradients.
• Formulate Adagrad and Adam with a common update rule and a simplified Adam variant by dropping the m_n corrective term (Equation 5).
• Derive convergence bounds for the non-convex setting by analyzing the expected squared gradient norm at a random iterate τ (defined with a weighting dependent on β1).
• Establish key lemmas that bound descent-direction deviation (Lemma 5.1) and the cumulative effect of momentum through a log-type sum (Lemma 5.2).
• Prove Theorems 1–2 for no-momentum cases and Theorems 3–4 for momentum, including the dependency on dimension d, gradient bound R, and smoothness L.
• Discuss optimal finite-horizon behavior and the equivalence of Adam and Adagrad under certain parameter regimes.

Experimental results

Research questions

• RQ1 Do Adagrad and Adam converge to a critical point for smooth, possibly non-convex objectives with bounded gradients?
• RQ2 What is the explicit bound on the expected squared gradient norm along the trajectory, and how does it depend on problem constants (dimension, gradient bound, smoothness) and optimizer parameters?
• RQ3 How does momentum (β1) affect convergence rates and constants, and can these dependencies be tightened compared to previous results?
• RQ4 Under what parameter settings do Adam and Adagrad achieve the same convergence rate, and how do default Adam parameters influence convergence in practice?
• RQ5 Can a simplified variant of Adam (dropping certain corrective terms) still guarantee convergence with a clear rate?

Key findings

• Convergence to a critical point is established for Adagrad and Adam in the non-convex, smooth setting with bounded gradients, with an explicit bound on the expected squared gradient norm.
• Adagrad achieves the standard O(log N / sqrt(N)) rate for the averaged gradient norm across iterations, holding for all step sizes.
• Adam achieves the same rate under appropriate choices of step sizes and decay parameters, and can converge without AMSGrad.
• The dependency of the convergence bound on the heavy-ball momentum decay rate β1 is improved from O((1−β1)−3) or O((1−β1)−5) in prior work to O((1−β1)−1).
• With momentum, increasing β1 deteriorates bounds, but the unified analysis shows near-equivalent asymptotics to Adagrad in certain regimes, explaining practical momentum benefits.
• The analysis also highlights that, at finite horizons, Adam and Adagrad are effectively twins under matched parameter scalings (α ~ N^−1/2, β2 ~ 1 − 1/N).