[Paper Review] Does Adam optimizer keep close to the optimal point?

Kiwook Bae, Heechang Ryu+1 more|arXiv (Cornell University)|Nov 1, 2019

Stochastic Gradient Optimization Techniques11 references18 citations

TL;DR

This paper demonstrates that Adam fails to stay near the optimal point in one-point convex regions when the effective learning rate becomes too large due to small second-moment estimates. To address this, the authors propose AdaFix, which fixes the second-moment estimate after it stabilizes, ensuring convergence and stability near optimal solutions, outperforming Adam and other adaptive methods on MNIST and CIFAR-10.

ABSTRACT

The adaptive optimizer for training neural networks has continually evolved to overcome the limitations of the previously proposed adaptive methods. Recent studies have found the rare counterexamples that Adam cannot converge to the optimal point. Those counterexamples reveal the distortion of Adam due to a small second momentum from a small gradient. Unlike previous studies, we show Adam cannot keep closer to the optimal point for not only the counterexamples but also a general convex region when the effective learning rate exceeds the certain bound. Subsequently, we propose an algorithm that overcomes Adam's limitation and ensures that it can reach and stay at the optimal point region.

Motivation & Objective

• To identify why Adam fails to remain near optimal points despite convergence in some cases.
• To analyze the instability of Adam in one-point convex regions where the optimal solution is locally convex.
• To derive theoretical bounds on the second-moment estimate that prevent divergence from optimal regions.
• To design a new adaptive optimizer, AdaFix, that ensures convergence and stability near optimal points.
• To validate AdaFix's superiority over Adam, AMSGrad, and AdaBound on standard vision benchmarks.

Proposed method

• The authors analyze Adam's behavior under the one-point convexity assumption, where the gradient consistently points toward the optimal point.
• They derive theoretical bounds on the second-moment estimate $\hat{v}_t$ to prevent the effective learning rate $\eta / \sqrt{\hat{v}_t}$ from becoming too large.
• AdaFix fixes $\hat{v}_t$ once it reaches a stable, sufficiently small value, preventing explosive learning rates due to small gradients.
• The algorithm approximates the L-smoothness constant $L$ along the training trajectory using the ratio of gradient differences to parameter updates.
• The fixed $\hat{v}_t$ is used to maintain a stable effective learning rate, avoiding both divergence and underfitting.
• AdaFix is evaluated on MNIST and CIFAR-10 using multiple architectures, comparing against Adam, AMSGrad, AdaBound, and SGDM.

Experimental results

Research questions

• RQ1 Can Adam maintain proximity to the optimal point in a one-point convex region, even when it reaches it?
• RQ2 What causes Adam to diverge from optimal regions despite convergence in theory?
• RQ3 What theoretical bounds on the second-moment estimate prevent Adam from escaping optimal regions?
• RQ4 How can the adaptive learning rate be stabilized to ensure both convergence and stability near optimal points?
• RQ5 Can a modified adaptive optimizer outperform Adam and its variants in standard vision benchmarks?

Key findings

• AdaFix achieves a test accuracy of 98.75% ± 0.02 on MNIST with a two-layer MLP, outperforming Adam (98.70% ± 0.03).
• On CIFAR-10 without data augmentation, AdaFix achieves 90.25% ± 0.19 on DenseNet53, surpassing Adam (90.07% ± 0.04).
• With data augmentation, AdaFix reaches 93.51% ± 0.17 on DenseNet53, matching or exceeding SGDM and other adaptive methods.
• Figure 2 shows that Adam's $||v||_2$ becomes very small, causing explosive effective learning rates and oscillatory test accuracy, while AdaFix stabilizes the learning rate.
• AMSGrad prevents learning rate explosion but suffers from overly small effective learning rates, whereas AdaFix balances stability and efficiency.
• The theoretical analysis shows that when $\delta \ll L$, the effective learning rate bound becomes approximately $L\eta$, justifying the need to fix $\hat{v}_t$ once it stabilizes.