[Paper Review] AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

Juntang Zhuang, Tommy Tang+5 more|arXiv (Cornell University)|Oct 14, 2020

Generative Adversarial Networks and Image SynthesisComputer Science219 citations

TL;DR

AdaBelief adapts per-parameter stepsizes by measuring belief in the observed gradient, achieving fast convergence, good generalization, and training stability.

ABSTRACT

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability.We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer. Code is available at https://github.com/juntang-zhuang/Adabelief-Optimizer

Motivation & Objective

• Motivate combining fast convergence of adaptive methods with good generalization and stability in challenging settings like GANs.
• Propose AdaBelief as an Adam-derived optimizer that adjusts stepsizes using gradient prediction error.
• Provide theoretical convergence analysis in convex and non-convex regimes.
• Empirically validate AdaBelief on image classification, language modeling, and GANs showing improved performance and stability.

Proposed method

• Define AdaBelief as a modification of Adam where the update uses m_t / sqrt(s_t) rather than m_t / sqrt(v_t).
• Here m_t is the EMA of gradients and v_t is the EMA of g_t^2, while s_t is the EMA of (g_t - m_t)^2.
• Bias-correct both m_t and s_t and perform parameter updates with projection onto a convex set using the sqrt of the bias-corrected s_t (plus epsilon).
• Interpret 1/sqrt(s_t) as the “belief” in the current gradient observation, increasing steps when the observation agrees with the prediction and decreasing steps when it deviates.
• Provide intuition and visual examples showing AdaBelief captures curvature information and differentiates updates by gradient sign and magnitude.
• Offer convergence analyses for both convex and non-convex stochastic optimization cases.

Experimental results

Research questions

• RQ1 Does AdaBelief retain fast convergence from adaptive methods while improving generalization akin to SGD?
• RQ2 Can AdaBelief provide training stability in GANs and other challenging settings while maintaining competitive accuracy?
• RQ3 How does AdaBelief perform in convex and non-convex optimization theoretically and empirically?
• RQ4 What practical benefits arise from using the belief-based scaling in real-world tasks like image classification and language modeling?

Key findings

• AdaBelief achieves fast convergence comparable to Adam and generalization similar to SGD in image classification tasks.
• AdaBelief demonstrates better training stability and sample quality than Adam in GAN training across CIFAR-10 with both small and larger generators.
• In ImageNet, AdaBelief attains comparable accuracy to SGD when using Adam-style defaults, reducing the generalization gap seen with some adaptive methods.
• Across language modeling, AdaBelief yields improved perplexities over competing optimizers.
• Across GAN benchmarks (WGAN, WGAN-GP), AdaBelief achieves lower FID scores than several baselines, indicating higher image fidelity and diversity.