[Paper Review] LaProp: a Better Way to Combine Momentum with Adaptive Gradient

Liu Ziyin, Zhikang T. Wang+1 more|arXiv (Cornell University)|Feb 12, 2020

Stochastic Gradient Optimization Techniques17 references15 citations

TL;DR

LaProp proposes a novel optimization method that decouples momentum and adaptivity in Adam-style optimizers, resolving instability caused by their coupling. By enabling independent hyperparameter tuning and interpolation between signed and adaptive gradient methods, LaProp achieves superior training speed and stability across diverse tasks, with a regret bound that demonstrates theoretical advantages over Adam.

ABSTRACT

We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.

Motivation & Objective

• To identify and resolve the instability in Adam-style optimizers caused by the unintended coupling between momentum and adaptivity.
• To decouple momentum and adaptivity to allow independent hyperparameter tuning and greater methodological flexibility.
• To enable a smooth interpolation between signed gradient methods and adaptive gradient methods through the decoupled framework.
• To demonstrate improved training speed and stability over Adam across multiple machine learning tasks.
• To provide a theoretical regret bound for LaProp on convex problems, showing a key advantage over Adam.

Proposed method

• LaProp introduces a new update rule that separates the momentum term from the adaptive learning rate computation.
• It maintains a separate exponential moving average for the gradient and its square, decoupling their update mechanisms.
• The method uses a modified learning rate schedule that allows independent control over momentum strength and adaptive scaling.
• LaProp enables interpolation between sign-based and adaptive gradient methods by adjusting the relative influence of momentum and adaptive components.
• The algorithm is designed to be backward compatible with existing Adam-like frameworks while improving convergence behavior.
• Theoretical analysis bounds the regret of LaProp on convex problems, highlighting a key difference from Adam's regret bound.

Experimental results

Research questions

• RQ1 How does the coupling between momentum and adaptivity in Adam-style optimizers affect training stability and convergence?
• RQ2 Can momentum and adaptivity be decoupled to improve optimization performance and hyperparameter flexibility?
• RQ3 What is the impact of decoupling on training speed and stability across diverse machine learning tasks?
• RQ4 How does the regret bound of LaProp compare to that of Adam in convex optimization settings?
• RQ5 Can LaProp achieve a smooth interpolation between sign-based and adaptive gradient methods?

Key findings

• LaProp consistently improves training speed and stability over Adam across a variety of machine learning tasks.
• The decoupling of momentum and adaptivity allows for greater flexibility in hyperparameter selection without compromising convergence.
• LaProp enables a smooth interpolation between signed gradient methods and adaptive gradient methods through its design.
• Theoretical analysis shows that LaProp's regret bound differs from Adam's by a key factor, indicating a fundamental advantage in convergence guarantees.
• Empirical results demonstrate that LaProp avoids divergence issues that arise in Adam when momentum and adaptivity parameters are mismatched.
• LaProp maintains strong performance across diverse tasks, suggesting broad applicability beyond specific architectures or datasets.