[论文解读] On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization

Xiangyi Chen, Sijia Liu+2 more|arXiv (Cornell University)|Aug 8, 2018

Stochastic Gradient Optimization Techniques参考文献 32被引用 94

一句话总结

本文在非凸随机优化中给出一组温和、可验证的条件，保证广义的 Adam 型自适应梯度方法收敛到一阶驻点，并引入 AdaFom 和带常数动量的变体，且给出证明的收敛速率。

ABSTRACT

This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam, AMSGrad and AdaGrad. Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these methods can achieve the convergence rate of order $O(\log{T}/\sqrt{T})$ for nonconvex stochastic optimization. We show the conditions are essential in the sense that violating them may make the algorithm diverge. Moreover, we propose and analyze a class of (deterministic) incremental adaptive gradient algorithms, which has the same $O(\log{T}/\sqrt{T})$ convergence rate. Our study could also be extended to a broader class of adaptive gradient methods in machine learning and optimization.

研究动机与目标

• Motivate the study of Adam-type adaptive gradient methods for non-convex optimization and identify gaps between practice and theory.
• Develop a unified analysis framework that yields convergence guarantees under mild conditions for a broad class of Adam-type algorithms.
• Introduce AdaFom (AdaGradient with First Order Momentum) and establish convergence with constant momentum.
• Provide practical, checkable sufficient conditions to monitor convergence in real algorithms.
• Show tightness of the rate bounds and illustrate implications for step-size oscillations and effective stepsizes.

提出的方法

• Model the update as a generalized Adam with m_t (first moment) and hat{v}_t (adaptive second moment) in x_{t+1}=x_t - alpha_t m_t / sqrt(hat{v}_t).
• Derive Assumptions A1–A3 (L-smooth f, bounded gradients/noisy gradients, unbiased noise with independence).
• Prove Theorem 3.1: convergence bound on the sum of inner products involving gradients, yielding min_t E||∇f(x_t)||^2 = O(s1(T)/s2(T)) given control of the effective stepsize gamma_t.
• Analyze the roles of Term A (gradient-scale squared norm) and Term B (oscillation of effective stepsizes) in the bound; show they govern convergence and possible divergence.
• Present corollaries for AMSGrad and AdaFom with specific step-size (e.g., alpha_t = 1/√t) and constant momentum, yielding O(1/√T) type convergence up to log factors.
• Provide examples demonstrating necessity of the conditions and the divergence risks if violated.
• Link theory to practice with MNIST and CIFAR-10 experiments comparing AMSGrad, Adam, AdaFom, and AdaGrad.

实验结果

研究问题

• RQ1 Under what mild conditions do Adam-type methods converge for non-convex stochastic optimization?
• RQ2 How do the adaptive second-moment estimate and momentum terms affect convergence and the role of oscillations in effective stepsizes?
• RQ3 Can variants like AdaFom and constant-momentum AMSGrad achieve convergence where Adam may fail?
• RQ4 What practical, checkable criteria can practitioners use to certify convergence or monitor progress?
• RQ5 How do empirical results on standard benchmarks reflect the theoretical convergence guarantees?

主要发现

• A broad Adam-type family converges to first-order stationary points under mild, verifiable conditions on stepsizes and parameters.
• The convergence rate is of order O(log T / sqrt(T)) for non-convex stochastic optimization under the proposed framework.
• AdaFom, which adds momentum only to the first moment while keeping AdaGrad’s second moment, converges whereas vanilla Adam can diverge in some cases.
• AMSGrad with constant momentum is proven to converge in non-convex settings, clarifying prior discrepancies between theory and practice.
• The sufficient conditions are practical to check and can help monitor convergence during training.
• Empirical results on MNIST and CIFAR-10 show AMSGrad/Adam perform similarly, AdaFom improves over AdaGrad, and overall trends align with the theoretical insights.