[Paper Review] Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients

Lukas Balles, Philipp Hennig|arXiv (Cornell University)|May 22, 2017

Architecture and Computational DesignEngineering78 citations

TL;DR

The paper decomposes Adam into two components—sign-based update direction and variance-adaptive magnitude—and analyzes them separately, proposing svag and m-svag as alternatives, with experiments showing sign dominates performance and variance adaptation aids robustness.

ABSTRACT

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an estimate of their relative variance. We disentangle these two aspects and analyze them in isolation, gaining insight into the mechanisms underlying ADAM. This analysis also extends recent results on adverse effects of ADAM on generalization, isolating the sign aspect as the problematic one. Transferring the variance adaptation to SGD gives rise to a novel method, completing the practitioner's toolbox for problems where ADAM fails.

Motivation & Objective

• Motivate and understand why Adam’s performance varies across problems by isolating the sign-based update from variance adaptation.
• Provide a principled derivation of element-wise variance adaptation factors.
• Demonstrate how applying variance adaptation to different update directions affects convergence and generalization.
• Propose practical implementations of variance-adapted methods, including m-svag, and compare them to Adam and sign-based methods.

Proposed method

• Interpret Adam as two components: sign-based update direction and variance-adapted update magnitude.
• Derive optimal variance-adaptation factors for sign-based updates and for gradient updates.
• Introduce Stochastic Variance-Adapted Gradient (SVAG) and its momentum variant (m-svag) and show their convergence properties under standard assumptions.
• Provide practical variance estimation by relating v_t and m_t as estimates of gradient moments and correct for bias in variance estimates.
• Present algorithm pseudocode for m-svag and discuss its hyperparameters and implementation details.

Experimental results

Research questions

• RQ1 Does separating the sign-based update from variance adaptation improve understanding of Adam’s behavior in stochastic optimization?
• RQ2 Can variance adaptation be derived as an optimal per-coordinate scaling factor for stochastic updates?
• RQ3 How do SVAG and m-svag perform compared to Adam and sign-based/non-sign-based baselines across varied tasks?
• RQ4 What is the impact of variance adaptation on generalization in adaptive methods?
• RQ5 How can variance adaptation be practically estimated and integrated with momentum in optimization routines?

Key findings

• The sign component is the dominant factor in Adam’s performance across most tested problems, though its usefulness is problem-dependent.
• Variance adaptation improves performance and can be applied to any update direction, not just Adam, as shown by m-svag.
• Optimal per-coordinate variance factors can be derived and align with the factors implicitly used by Adam, especially for sign-descent variants.
• m-svag avoids some generalization harms associated with sign-based adaptive methods while maintaining or improving training performance.
• Sign-based methods can outperform non-sign-based methods on some tasks, but may harm generalization on others, depending on problem structure.
• Experimental results on four problems show clustering of sign-based vs non-sign-based methods, with variance adaptation providing further gains.