[Paper Review] Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Liangchen Luo, Yuanhao Xiong+2 more|arXiv (Cornell University)|Feb 26, 2019

Stochastic Gradient Optimization Techniques189 citations

TL;DR

The paper introduces AdaBound and AMSBound, dynamic-bound variants of Adam/AMSGrad that start as adaptive optimizers and gradually transition to SGD, with convergence guarantees and improved generalization across tasks.

ABSTRACT

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at https://github.com/Luolc/AdaBound .

Motivation & Objective

• Motivate the limitations of adaptive optimizers like Adam/AMSGrad in generalization and convergence.
• Propose learning-rate bound mechanisms that transition from adaptive behavior to SGD over time.
• Provide theoretical convergence guarantees in the convex setting for the new methods.
• Demonstrate empirical benefits on computer vision and natural language processing tasks across various architectures.

Proposed method

• Formulate AdaBound by clipping per-parameter learning rates with time-evolving lower and upper bounds that converge to a final step size.
• Define eta_l(t) and eta_u(t) to create a gradual transform from Adam/AMSGrad to SGD(M).
• Prove regret bounds and convergence properties for AdaBound (and AMSBound) under convex assumptions.
• Compare AdaBound/AMSBound to Sgd(M), AdaGrad, Adam, and AMSGrad through experiments on MNIST, CIFAR-10, and Penn Treebank.
• Provide implementation details and discuss hyperparameter choices and bound schedules.

Experimental results

Research questions

• RQ1 Can dynamic bound learning-rate schedules prevent extreme updates and improve generalization for adaptive optimizers?
• RQ2 Do AdaBound and AMSBound retain fast initial convergence while achieving SGD-like generalization?
• RQ3 What are the theoretical guarantees (convergence/regret) for these bound-based adaptive methods in convex settings?
• RQ4 How do the proposed methods perform across diverse architectures and tasks (vision and NLP) compared to baseline optimizers?
• RQ5 Is there a practical, tunable bound schedule that works well without extensive hyperparameter tuning?

Key findings

• AdaBound/AMSBound achieve fast early training similar to adaptive methods and converge with strong generalization comparable to or better than SGD/M.
• The dynamic bounds ensure a smooth transition from adaptive behavior to SGD, mitigating issues from extreme learning rates.
• Theoretical analysis provides regret bounds O(sqrt(T)) and convergence guarantees under convexity.
• Empirical results across MNIST, CIFAR-10, and Penn Treebank show improved test accuracy and perplexity over Adam/AMSGrad and competitive results with SGD(M).
• Complex models (DenseNet, ResNet, multi-layer LSTM) exhibit larger gains, highlighting benefits in deeper architectures.