[Paper Review] On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

Xiangyi Chen, Sijia Liu+2 more|arXiv (Cornell University)|Aug 8, 2018

Stochastic Gradient Optimization Techniques117 citations

TL;DR

The paper provides a unifying convergence framework for Adam-type adaptive gradient methods in non-convex optimization, establishing conditions under which these methods converge to stationary points with a rate of O(log T / sqrt(T)). It also introduces AdaFom and analyzes constant-momentum variants like AMSGrad and AdaFom.

ABSTRACT

This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these methods can achieve the convergence rate of order $O(\\log{T}/\\sqrt{T})$ for nonconvex stochastic optimization. We show the conditions are essential in the sense that violating them may make the algorithm diverge. Moreover, we propose and analyze a class of (deterministic) incremental adaptive gradient algorithms, which has the same $O(\\log{T}/\\sqrt{T})$ convergence rate. Our study could also be extended to a broader class of adaptive gradient methods in machine learning and optimization.

Motivation & Objective

• Motivate the study of adaptive gradient methods for non-convex optimization and understand convergence guarantees.
• Develop a general Ada-typed algorithmic framework that encompasses Adam, AMSGrad, AdaGrad, AdaFom, and SGD variants.
• Derive mild, practical conditions on stepsizes and moment parameters that ensure convergence to stationary points with a sublinear rate.
• Introduce AdaFom (AdaGrad with First Order Momentum) and show its convergence properties.
• Show the sharpness of the conditions by exhibiting potential divergence when violated.
• Demonstrate applicability to constant momentum settings and finite-sum problems.

Proposed method

• Propose a generalized Adam-type update with m_t = β1,t m_{t-1} + (1−β1,t) g_t and an adaptive 〈hat{v}〉_t = h_t(g_1,...,g_t).
• Define the effective stepsize as α_t / sqrt(〈hat{v}〉_t) and analyze its oscillation impacts on convergence.
• Establish a main theorem giving a bound on the sum of gradient inner products and relate it to two term-growths A and B (Term A and Term B).
• Derive a convergence rate for E[min_{t∈[T]} ||∇f(x_t)||^2] = O(s1(T)/s2(T)) with s1(T) = o(s2(T)).
• Provide corollaries for AMSGrad and AdaFom with α_t = 1/√t and show sublinear rates with log T factors.
• Discuss how adding momentum to only the first moment (AdaFom) fixes AdaGrad-like divergence tendencies.

Experimental results

Research questions

• RQ1 Under what mild conditions on stepsizes and momentum parameters do Adam-type algorithms converge to first-order stationary points in non-convex settings?
• RQ2 How does the oscillation of effective stepsizes influence convergence and rate for AdaGrad/Adam-type methods?
• RQ3 Can variants like AdaFom and constant-momentum AMSGrad achieve convergence, and what are their rates?
• RQ4 What practical criteria can practitioners use to certify convergence or monitor progress of Adam-type methods?
• RQ5 Is the observed non-convergence of Adam due to Term A or Term B in the theoretical framework?

Key findings

• A general Adam-type convergence framework is established under mild assumptions, yielding a convergence rate of O(log T / sqrt(T)).
• AdaFom is shown to converge under standard assumptions, whereas vanilla Adam can diverge under certain conditions.
• AMSGrad with constant momentum also converges in non-convex settings, clarifying its behavior relative to Adam.
• The analysis identifies two critical components: Term A (gradient magnitude accumulation) and Term B (oscillation of effective stepsizes), which govern convergence and can explain practical performance differences.
• Corollaries show AMSGrad and AdaFom achieve sublinear rates with α_t = 1/√t, matching known rates up to a log T factor.
• The conditions provided are tight and offer practical monitoring tools for convergence in real-world training.