[Paper Review] On Empirical Comparisons of Optimizers for Deep Learning

Dami Choi, Christopher J. Shallue+4 more|arXiv (Cornell University)|Oct 11, 2019

Advanced Neural Network Applications57 references185 citations

TL;DR

The paper shows that hyperparameter tuning protocols drive optimizer rankings, and inclusion relationships between optimizers (e.g., adaptive methods vs. momentum) reliably predict comparative performance. It argues that well-tuned adaptive methods never underperform momentum or SGD.

ABSTRACT

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

Motivation & Objective

• Assess how hyperparameter tuning protocols influence optimizer rankings in deep learning.
• Investigate whether inclusion relationships between optimizers hold under realistic tuning budgets.
• Identify practical hyperparameter tuning considerations that affect fair benchmarking of optimizers.

Proposed method

• Experimentally compare optimizers under varied hyperparameter search spaces.
• Assess inclusion relationships between optimizers (adaptive methods, momentum, SGD).
• Analyze how tuning effort impacts relative performance of optimizers.

Experimental results

Research questions

• RQ1 Do hyperparameter search spaces determine the relative performance of optimizers in deep learning?
• RQ2 Do inclusion relationships among optimizers hold under practical tuning budgets?
• RQ3 Can well-tuned adaptive methods underperform momentum or SGD, or do they always approximate them?
• RQ4 What practical hyperparameter tuning guidelines are needed for fair benchmarks?

Key findings

• Hyperparameter tuning protocol critically shapes optimizer rankings.
• Changing hyperparameter search spaces can invert conclusions of comparative studies.
• Adaptive gradient methods do not underperform momentum or SGD when tuning is allowed.
• Inclusion relationships between optimizers reliably predict practical comparisons.
• The paper provides practical tips for tuning adaptive gradient methods and cautions on benchmarking practices.