[Paper Review] The Marginal Value of Adaptive Gradient Methods in Machine Learning

Ashia Wilson, Rebecca Roelofs+3 more|arXiv (Cornell University)|May 23, 2017

Stochastic Gradient Optimization TechniquesComputer Science20 references552 citations

TL;DR

Adaptive gradient methods (AdaGrad, RMSProp, Adam) often converge to very different solutions than GD/SGD and generalize worse in overparameterized settings, while SGD/HB generally outperform them in test performance even with similar training loss; tuning can significantly affect Adam’s performance.

ABSTRACT

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Motivation & Objective

• Motivate and understand how optimization method choices affect generalization in deep learning.
• Demonstrate that adaptive methods can converge to solutions with poor out-of-sample performance in a constructed binary classification setting.
• Empirically compare adaptive vs non-adaptive methods across state-of-the-art deep learning models and tasks.
• Provide practical guidance on tuning learning rates and decay schemes to improve performance across methods.

Proposed method

• Formulate optimization with adaptive metric H_k and derive updates for AdaGrad, RMSProp, and Adam (diagonal H_k).
• Show that in a simple least-squares binary classification, non-adaptive methods converge to the minimum-norm (max-margin) solution within the row span of X, while adaptive methods converge to a solution proportional to sign(X^T y) under certain conditions (Lemma 3.1).
• Construct an infinite-dimensional generative model where adaptive methods attain test error approaching 1/2 while SGD achieves zero test error.
• Conduct extensive deep learning experiments (CIFAR-10, War and Peace language model, Penn Treebank parsing) comparing SGD, Heavy Ball, AdaGrad, RMSProp, and Adam with careful hyperparameter tuning.
• Propose a practical scheme for tuning learning rates and decays that performs well across tasks.

Experimental results

Research questions

• RQ1 Do adaptive gradient methods converge to minimizers that generalize differently from those found by non-adaptive methods?
• RQ2 Can adaptive methods generalize worse than SGD/SGD with momentum even when training performance is similar or better?
• RQ3 How much hyperparameter tuning is required for adaptive methods to perform well in deep learning tasks?
• RQ4 What practical tuning strategies improve Adam's performance across diverse deep learning tasks?

Key findings

• Adaptive methods often find solutions with worse generalization than non-adaptive methods, even when training performance is comparable or better.
• Across evaluated models and tasks, SGD/SGD with momentum outperform adaptive methods on development/test sets given the same tuning effort.
• Adaptive methods tend to show faster initial training progress but plateau early on development/test performance.
• Tuning initial learning rate and decay scheme for Adam yields significant improvements over its default settings in all cases.
• In a simple constructed binary classification problem, AdaGrad/Adam/RMSProp converge to solutions with poor generalization while SGD achieves zero test error.
• Empirical results on CIFAR-10, War and Peace, and Penn Treebank indicate adaptive methods generalize worse than SGD/HB, sometimes substantially so, despite training losses.
• The authors provide a practical learning-rate tuning scheme that improves performance across methods.