[Paper Review] No More Pesky Learning Rates

Tom Schaul, Sixin Zhang+1 more|arXiv (Cornell University)|Jun 6, 2012

Stochastic Gradient Optimization Techniques21 references291 citations

TL;DR

This paper proposes a novel adaptive learning rate method for stochastic gradient descent (SGD) that automatically adjusts learning rates based on local gradient variance and curvature estimates, eliminating the need for manual tuning. The approach dynamically increases or decreases learning rates, achieving performance comparable to optimally tuned SGD across convex and non-convex tasks without hyperparameter sensitivity.

ABSTRACT

The performance of stochastic gradient descent (SGD) depends critically on how learning rates are tuned and decreased over time. We propose a method to automatically adjust multiple learning rates so as to minimize the expected error at any one time. The method relies on local gradient variations across samples. In our approach, learning rates can increase as well as decrease, making it suitable for non-stationary problems. Using a number of convex and non-convex learning tasks, we show that the resulting algorithm matches the performance of SGD or other adaptive approaches with their best settings obtained through systematic search, and effectively removes the need for learning rate tuning.

Motivation & Objective

• To eliminate the need for manual learning rate tuning in stochastic gradient descent (SGD) across diverse machine learning tasks.
• To develop an adaptive learning rate strategy that can increase or decrease dynamically, making it suitable for non-stationary and online learning problems.
• To derive a theoretically grounded formula for optimal learning rates that minimizes expected loss after each update.
• To implement the method with minimal hyperparameters, ensuring robustness and stability across different model architectures and data dimensions.

Proposed method

• The method derives optimal learning rates under an idealized quadratic, separable loss model, using estimates of gradient variance and local curvature.
• It formulates learning rates as a function of the ratio of squared gradient means to variances, enabling automatic adaptation per parameter or globally.
• The algorithm uses exponential moving averages to estimate gradient moments, avoiding the need for pre-defined annealing schedules.
• Five variants are proposed: global, block-wise, element-wise, and two hybrid forms, all requiring no manual tuning.
• A critical initialization parameter $ C $ is introduced to prevent instability, with a heuristic $ C = d/10 $ ensuring robustness across dimensions.
• The method is implemented via a meta-optimization framework where learning rates are updated multiplicatively based on gradient direction and magnitude.

Experimental results

Research questions

• RQ1 Can learning rate adaptation be automated such that no manual tuning is required for SGD across diverse learning tasks?
• RQ2 How can learning rates be adjusted dynamically to respond to both stationary and non-stationary data distributions?
• RQ3 What is the optimal learning rate schedule that minimizes expected loss after each update, given local gradient statistics?
• RQ4 Can the method achieve performance comparable to the best-tuned SGD without relying on fixed hyperparameters?
• RQ5 How sensitive is the algorithm to initialization parameters, and can a robust heuristic be derived for safe default settings?

Key findings

• The proposed method achieves performance on par with optimally tuned SGD across a range of convex and non-convex tasks, including MNIST and XOR classification.
• The algorithm is insensitive to the choice of the initialization parameter $ C $, with stable performance across 8 orders of magnitude, validating the heuristic $ C = d/10 $.
• For non-stationary problems, learning rates increase when data shifts, unlike AdaGrad, which monotonically decreases rates.
• The method outperforms or matches other adaptive methods like AdaGrad and SMD, without requiring tuning of learning rate or meta-parameters.
• Empirical results show that the algorithm remains stable and effective even with very large model dimensions, up to 500,000 neurons in a single-layer MLP.
• The theoretical derivation of optimal learning rates based on gradient variance and curvature leads to practical algorithms that converge efficiently without pre-specified decay schedules.