[Paper Review] Online Learning Rate Adaptation with Hypergradient Descent

Atılım Güneş Baydin, Robert Cornish+3 more|arXiv (Cornell University)|Mar 14, 2017

Stochastic Gradient Optimization Techniques27 references77 citations

TL;DR

The paper introduces hypergradient descent to adapt the global learning rate online, improving convergence for SGD, SGD with Nesterov momentum, and Adam while reducing manual learning-rate tuning.

ABSTRACT

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov momentum, and Adam, showing that it significantly reduces the need for the manual tuning of the initial learning rate for these commonly used algorithms. Our method works by dynamically updating the learning rate during optimization using the gradient with respect to the learning rate of the update rule itself. Computing this "hypergradient" needs little additional computation, requires only one extra copy of the original gradient to be stored in memory, and relies upon nothing more than what is provided by reverse-mode automatic differentiation.

Motivation & Objective

• Motivate the need for automatic learning-rate adaptation in gradient-based optimizers.
• Propose a general, computation- and memory-efficient method to update the learning rate online using hypergradients.
• Demonstrate the method by applying hypergradient descent to SGD, SGDN, and Adam across standard optimization problems.
• Show that hypergradient descent reduces the dependence on the initial learning rate and accelerates convergence.

Proposed method

• Define hypergradient descent by performing gradient descent on the learning rate using the derivative of the objective with respect to the learning rate.
• Derive the basic HD update: α_t = α_{t-1} - β ∂f(θ_{t-1})/∂α and θ_t = θ_{t-1} - α_t ∇f(θ_{t-1}), where the hypergradient ∂f(θ_{t-1})/∂α = ∇f(θ_{t-1}) · ( -∇f(θ_{t-2}) ).
• Compute the hypergradient using a single extra gradient copy and a dot product, incurring minimal memory and computation overhead.
• Instantiate HD variants for SGD (SGD-HD), SGD with Nesterov momentum (SGDN-HD), and Adam (Adam-HD), including both additive and multiplicative hypergradient update forms.
• Provide implementation mappings: SGD-HD, SGDN-HD, Adam-HD correspond to replacing the underlying update statements in the regular algorithms with hypergradient-based learning-rate updates.
• Discuss potential extensions (transition to fixed α∞, higher-order hypergradients) and empirical evaluation setup.

Experimental results

Research questions

• RQ1 Does online learning-rate adaptation via hypergradients improve convergence across common gradient-based optimizers?
• RQ2 Are SGD, SGDN, and Adam with hypergradient descent less sensitive to the initial learning rate α_0?
• RQ3 How does HD affect training and validation performance on neural networks compared to their non-HD counterparts?
• RQ4 What are practical considerations (memory, computation, hypergradient learning rate β) for applying HD in large-scale settings?

Key findings

• HD variants consistently improve or match the performance of their non-HD counterparts across logistic regression, a multilayer network on MNIST, and a VGG-like network on CIFAR-10.
• The learning rate α_t typically rises initially and then decays toward a small value, enabling auto-geometry adaptation.
• For a given untuned α_0, SGD-HD, SGDN-HD, and Adam-HD bring the loss trajectory closer to the optimal trajectory that would be achieved with a tuned α_0.
• Adam-HD often achieves notably better training and sometimes validation performance than standard Adam.
• HD reduces the need for extensive hyperparameter searches (grid/random/bayesian) to find effective learning rates.
• The approach is memory-efficient, requiring only one extra copy of the gradient and no additional automatic-differentiation machinery.