[Paper Review] A Novel Convergence Analysis for Algorithms of the Adam Family

Zhishuai Guo, Yi Xu+3 more|arXiv (Cornell University)|Dec 7, 2021

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper presents a novel, generic convergence analysis for Adam-family optimization algorithms, including Adam, AMSGrad, and Adabound. By requiring only a sufficiently large or increasing momentum parameter and bounded adaptive step sizes, the analysis establishes convergence for non-convex optimization under mild stochastic gradient conditions, while also proving a variance-diminishing property for gradient estimators.

ABSTRACT

Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam. To address theses issues, in this extended abstract, we present a simple and generic proof of convergence for a family of Adam-style methods (including Adam, AMSGrad, Adabound, etc.). Our analysis only requires an increasing or large "momentum" parameter for the first-order moment, which is indeed the case used in practice, and a boundness condition on the adaptive factor of the step size, which applies to all variants of Adam under mild conditions of stochastic gradients. We also establish a variance diminishing result for the used stochastic gradient estimators. Indeed, our analysis of Adam is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non-convex optimization problems, including min-max, compositional, and bilevel optimization problems. For the full (earlier) version of this extended abstract, please refer to arXiv:2104.14840.

Motivation & Objective

• To address the lack of rigorous and practical convergence guarantees for Adam and its variants in non-convex deep learning settings.
• To overcome the limitations of prior analyses that rely on strong or impractical assumptions about gradient updates or adaptive step size behavior.
• To unify the convergence analysis of multiple Adam-style algorithms under a single, simple framework.
• To establish conditions under which stochastic gradient estimators used in these methods exhibit variance reduction.
• To extend the applicability of convergence results beyond standard minimization to broader classes of optimization problems, such as min-max and bilevel problems.

Proposed method

• Propose a new convergence proof framework based on a momentum parameter that is increasing or sufficiently large, a condition commonly satisfied in practice.
• Introduce a boundedness condition on the adaptive factor of the step size, which holds for all major Adam variants under mild stochastic gradient assumptions.
• Leverage a variance-diminishing property of the stochastic gradient estimators used in the algorithm, ensuring stable convergence.
• Apply the analysis to a broad family of non-convex optimization problems, including min-max and compositional optimization.
• Use a unified mathematical structure to analyze multiple algorithms (e.g., Adam, AMSGrad, Adabound) without requiring algorithm-specific derivations.
• Establish convergence under minimal assumptions, avoiding the restrictive conditions found in prior theoretical works.

Experimental results

Research questions

• RQ1 Can a unified and simple convergence analysis be developed for Adam-family algorithms that applies beyond the original Adam optimizer?
• RQ2 What minimal assumptions on the momentum and adaptive step size are sufficient to guarantee convergence in non-convex settings?
• RQ3 How can the variance of stochastic gradient estimators be controlled to support convergence in practice?
• RQ4 Can the proposed analysis be extended to non-convex problems beyond standard minimization, such as min-max and bilevel optimization?
• RQ5 Why do existing theoretical analyses fail to ensure convergence in practice, and how can this be remedied?

Key findings

• The proposed convergence analysis applies generically to Adam, AMSGrad, Adabound, and other variants under mild, practical assumptions.
• Convergence is guaranteed when the momentum parameter is sufficiently large or increasing, a condition typically met in real training setups.
• The adaptive step size factor remains bounded under mild conditions on stochastic gradients, ensuring stability in the update process.
• The stochastic gradient estimators used in the method exhibit a variance-diminishing property, which supports convergence.
• The analysis framework is general enough to be extended to non-convex problems such as min-max and bilevel optimization.
• The theoretical results are simpler and more practical than prior analyses, avoiding overly restrictive assumptions that limit real-world applicability.