[Paper Review] Lookahead Optimizer: k steps forward, 1 step back

Michael R. Zhang, James Lucas+2 more|arXiv (Cornell University)|Jul 19, 2019

Stochastic Gradient Optimization Techniques44 references381 citations

TL;DR

The Lookahead optimizer wraps any standard inner optimizer by updating fast weights for k steps and then updating slow weights once toward the fast weights, reducing variance and improving convergence with minimal overhead.

ABSTRACT

The vast majority of successful deep neural networks are trained using variants of stochastic gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly categorized into two approaches: (1) adaptive learning rate schemes, such as AdaGrad and Adam, and (2) accelerated schemes, such as heavy-ball and Nesterov momentum. In this paper, we propose a new optimization algorithm, Lookahead, that is orthogonal to these previous approaches and iteratively updates two sets of weights. Intuitively, the algorithm chooses a search direction by looking ahead at the sequence of fast weights generated by another optimizer. We show that Lookahead improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. We empirically demonstrate Lookahead can significantly improve the performance of SGD and Adam, even with their default hyperparameter settings on ImageNet, CIFAR-10/100, neural machine translation, and Penn Treebank.

Motivation & Objective

• Introduce Lookahead, a two-tier optimization method that integrates with existing optimizers.
• Show that Lookahead reduces variance and improves stability in neural network training.
• Demonstrate empirical gains across diverse tasks with minimal hyperparameter tuning.
• Analyze convergence properties and provide guidelines for selecting the slow weights step size α.

Proposed method

• Maintain slow weights φ and fast weights θ that are synchronized every k inner updates.
• Update fast weights θ using any standard optimizer A on mini-batches.
• After k inner updates, update slow weights φ toward θ via φ ← φ + α(θ − φ) and reset θ to φ.
• Provide a principled adaptively chosen or fixed α, with a quadratic-approximation-based justification.
• Show that slow weights follow an EMA of the fast weights within each inner loop.
• Discuss computational overhead as O((k+1)/k) relative to the inner optimizer and require one extra parameter copy.

Experimental results

Research questions

• RQ1 Does Lookahead improve convergence speed and stability when wrapped around standard optimizers like SGD or Adam?
• RQ2 How does the Lookahead slow weights step size α influence convergence and stability, both in theory and practice?
• RQ3 Is Lookahead robust to hyperparameter choices such as k and α across different architectures and tasks?
• RQ4 What are the convergence properties and variance reductions Lookahead provides in noisy and deterministic quadratic models?

Key findings

• Lookahead yields faster convergence and often better generalization when combined with SGD or Adam across CIFAR, ImageNet, language models, and machine translation.
• The slow weights update acts as an EMA of final fast weights, reducing variance and improving stability.
• Lookahead is robust to inner optimizer choices and hyperparameters, with fixed α performing well across tasks.
• In a noisy quadratic model, Lookahead’s steady-state variance is strictly lower than SGD’s for the same learning rate, given appropriate settings.
• Deterministic quadratic analysis shows Lookahead can improve convergence rates in under-damped regimes.
• Empirical results show Lookahead achieving competitive or superior final accuracies with minimal hyperparameter tuning.