[Paper Review] Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization

Aaron Defazio, Samy Jelassi|arXiv (Cornell University)|Jan 26, 2021

Stochastic Gradient Optimization Techniques27 references39 citations

TL;DR

MADGRAD introduces a momentumized, adaptive dual-averaging gradient method that combines AdaGrad-style adaptivity with momentum to achieve strong performance and convergence guarantees on convex problems, while delivering competitive deep learning results across vision and NLP tasks.

ABSTRACT

We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems from multiple fields, including classification and image-to-image tasks in vision, and recurrent and bidirectionally-masked models in natural language processing. For each of these tasks, MADGRAD matches or outperforms both SGD and ADAM in test set performance, even on problems for which adaptive methods normally perform poorly.

Motivation & Objective

• Motivate and design an optimization method that marries adaptivity with robust generalization for deep learning.
• Develop MADGRAD by building on dual averaging of AdaGrad and incorporating momentum and a cube-root denominator for stable, adaptive updates.
• Provide theoretical convergence guarantees under convexity and demonstrate empirical performance across vision and NLP tasks.

Proposed method

• Adopt dual averaging form of AdaGrad instead of the common mirror-descent formulation.
• Center the proximal function around the initial point to avoid divergent behavior in neural network training.
• Introduce a stage-wise, front-weighted gradient sequence with lambda_k scaling to preserve an effective step size.
• Incorporate momentum via a moving-average/inline-averaging scheme that aligns with momentum concepts in SGD variants.
• Use a cube-root denominator in the adaptive scaling to maintain proper effective step sizes and improve stability.
• Provide a convergence theorem under convexity with a bound that scales with key problem constants.

Experimental results

Research questions

• RQ1 Can MADGRAD achieve state-of-the-art or competitive test performance across standard deep learning benchmarks while preserving convergence guarantees?
• RQ2 How do dual averaging, momentum, and cube-root adaptive scaling interact to affect convergence and generalization in stochastic optimization?
• RQ3 What are the theoretical and empirical trade-offs between MADGRAD and established optimizers like Adam, AdaGrad, and SGD with momentum?

Key findings

• MADGRAD delivers matching or superior test-set performance relative to SGD and ADAM across various deep learning tasks.
• The method achieves strong convergence guarantees under convexity for the proposed updates.
• MADGRAD remains effective for sparse problems when momentum is not used, addressing a limitation of some adaptive methods.
• The cube-root denominator in the adaptive scaling helps preserve an appropriate effective step size and improves empirical stability.
• Compared to AMSGrad-like Adam bounds, MADGRAD avoids some of the problematic dependencies and log factors they exhibit.