[Paper Review] Quasi-hyperbolic momentum and Adam for deep learning

Jerry Ma, Denis Yarats|arXiv (Cornell University)|Oct 16, 2018

Stochastic Gradient Optimization Techniques45 references48 citations

TL;DR

Introduces Quasi-hyperbolic Momentum (QHM) and its Adam variant (QHAdam), simple two-term interpolations between plain SGD and momentum/Adam components, showing improved stability and performance across diverse deep learning tasks.

ABSTRACT

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

Motivation & Objective

• Motivate momentum-based acceleration for stochastic optimization and address variance reduction in SGD updates.
• Introduce QHM as a simple interpolation between plain SGD and momentum to control update staleness.
• Show that QHM recovers or closely relates to many existing optimization algorithms (e.g., NAG, PID, SNV, AccSGD) and characterize its recoverable algorithm set.
• Propose QHAdam as a QHM-based variant of Adam and demonstrate practical benefits across tasks.
• Provide practical guidelines and open-source code for practitioners to adopt QHM/QHAdam.

Proposed method

• Define QHM update: g_{t+1}  β g_t + (1-β) ∇L̂_t(θ_t) and θ_{t+1}  θ_t - α[(1-ν)∇L̂_t(θ_t) + ν g_{t+1}].
• Explain interpretation of ν as immediate discounting that blends SGD and momentum.
• Connect QHM to NAG, PID, SNV, AccSGD, Robust Momentum, and Triple Momentum, showing recoveries and relationships.
• Introduce QHAdam by replacing Adam’s moment estimators with quasi-hyperbolic terms, detailing update rules and showing it recovers Adam under ν1=ν2=1 and RMSProp/NAdam under other settings.
• Provide practical tuning guidelines and discuss convergence/robustness implications, supported by empirical experiments.

Experimental results

Research questions

• RQ1 Can QHM improve optimization efficiency and stability relative to traditional momentum and NAG across standard DL tasks?
• RQ2 How does the immediate discount factor ν influence the variance/staleness of momentum in stochastic settings?
• RQ3 What is the relationship between QHM and other two-state optimization algorithms, and can QHM recover them all efficiently?
• RQ4 Does the QHAdam variant offer practical benefits over Adam in terms of stability and performance across diverse domains?

Key findings

• QHM and QHAdam consistently improve training and validation performance across multiple tasks compared to vanilla momentum/NAG/Adam.
• A default configuration with ν=0.7 and β=0.999 often outperforms optimized parameterizations of NAG or Adam.
• In image translation (WMT16 EN-DE) QHAdam yields state-of-the-art BLEU of 29.45 and improved stability.
• Case studies show faster convergence or stronger performance in image recognition, language modeling, RL, and NMT when using QHM/QHAdam.
• QH algorithms are computationally inexpensive and conceptually simple, with practical tuning guidance provided.
• Code for QHM/QHAdam is released (qhoptim repository).