[Paper Review] FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizers by Exploiting Strong Convexity

Yangfan Zhou, Kaizhu Huang+4 more|arXiv (Cornell University)|Apr 28, 2021

Stochastic Gradient Optimization TechniquesComputer Science35 references16 citations

TL;DR

This paper proposes FastAdaBelief, a novel adaptive optimization algorithm that improves convergence speed in strongly convex and non-convex settings by exploiting strong convexity through a tailored step size and curvature-aware momentum. It achieves a data-dependent O(log T) regret bound—significantly faster than AdaBelief’s O(√T)—while maintaining superior generalization, outperforming all mainstream optimizers in training speed and test accuracy across image classification and language modeling benchmarks.

ABSTRACT

AdaBelief, one of the current best optimizers, demonstrates superior generalization ability compared to the popular Adam algorithm by viewing the exponential moving average of observed gradients. AdaBelief is theoretically appealing in that it has a data-dependent $O(\sqrt{T})$ regret bound when objective functions are convex, where $T$ is a time horizon. It remains however an open problem whether the convergence rate can be further improved without sacrificing its generalization ability. %on how to exploit strong convexity to further improve the convergence rate of AdaBelief. To this end, we make a first attempt in this work and design a novel optimization algorithm called FastAdaBelief that aims to exploit its strong convexity in order to achieve an even faster convergence rate. In particular, by adjusting the step size that better considers strong convexity and prevents fluctuation, our proposed FastAdaBelief demonstrates excellent generalization ability as well as superior convergence. As an important theoretical contribution, we prove that FastAdaBelief attains a data-dependant $O(\log T)$ regret bound, which is substantially lower than AdaBelief. On the empirical side, we validate our theoretical analysis with extensive experiments in both scenarios of strong and non-strong convexity on three popular baseline models. Experimental results are very encouraging: FastAdaBelief converges the quickest in comparison to all mainstream algorithms while maintaining an excellent generalization ability, in cases of both strong or non-strong convexity. FastAdaBelief is thus posited as a new benchmark model for the research community.

Motivation & Objective

• To address the open problem of whether AdaBelief’s convergence rate can be further improved without sacrificing generalization in strongly convex settings.
• To design an adaptive optimizer that leverages strong convexity to achieve faster convergence than existing belief-based adaptive methods.
• To theoretically prove a tighter regret bound for the proposed algorithm under strong convexity.
• To empirically validate the superiority of FastAdaBelief in both strongly convex and non-convex deep learning scenarios.

Proposed method

• Introduces a modified step size that explicitly accounts for strong convexity, reducing oscillations and accelerating convergence.
• Replaces the second-order momentum in AdaBelief with a curvature-aware, diagonal matrix that adapts to the Hessian-like structure of gradients.
• Employs a vanishing factor δ/t in the momentum update to better approximate the ideal step size and stabilize training.
• Derives a new data-dependent regret bound of O(log T) for strongly convex functions, improving upon AdaBelief’s O(√T).
• Designs a novel adaptive learning rate mechanism that combines belief in gradients with strong convexity information to balance convergence and generalization.
• Applies the algorithm to standard deep learning tasks using CNNs and LSTMs, with extensive ablation and comparison across multiple datasets.

Experimental results

Research questions

• RQ1 Can the convergence rate of AdaBelief be improved in strongly convex optimization without degrading generalization?
• RQ2 What is the theoretical regret bound of an adaptive optimizer that explicitly exploits strong convexity?
• RQ3 How does FastAdaBelief compare to SAdam and other Adam variants in terms of convergence speed and test performance?
• RQ4 Does the proposed algorithm maintain fast convergence in non-convex deep learning settings?

Key findings

• FastAdaBelief achieves a data-dependent regret bound of O(log T) in strongly convex settings, which is substantially tighter than AdaBelief’s O(√T).
• On CIFAR-10, FastAdaBelief converges faster than all baselines, reaching the highest test accuracy (96.0%) and lowest loss (0.15) after 200 epochs.
• In language modeling on Penn Treebank, FastAdaBelief achieves the lowest perplexity of 61.21 on 3-layer LSTM, outperforming Adam (64.71) and AdaBelief (61.23).
• On SVHN and CIFAR-100, FastAdaBelief converges the fastest in all experiments, with the lowest training loss across all iterations.
• Despite its faster convergence, FastAdaBelief maintains excellent generalization, matching or exceeding the performance of SGD and AdaBelief in test accuracy.
• The algorithm demonstrates robustness across diverse architectures (DenseNet-121, ResNet-34, VGG-11) and tasks, establishing itself as a new benchmark optimizer.