[Paper Review] An overview of gradient descent optimization algorithms

Sebastian Ruder|arXiv (Cornell University)|Sep 15, 2016

Stochastic Gradient Optimization Techniques18 references4,784 citations

TL;DR

A survey of gradient descent variants and popular optimization algorithms for training neural networks, with intuition on their behavior, strengths, and use-cases.

ABSTRACT

Gradient descent optimization algorithms, while increasingly popular, are often used as black-box optimizers, as practical explanations of their strengths and weaknesses are hard to come by. This article aims to provide the reader with intuitions with regard to the behaviour of different algorithms that will allow her to put them to use. In the course of this overview, we look at different variants of gradient descent, summarize challenges, introduce the most common optimization algorithms, review architectures in a parallel and distributed setting, and investigate additional strategies for optimizing gradient descent.

Motivation & Objective

• Explain the landscape of gradient descent variants and their practical implications for training neural networks.
• Summarize the challenges in training with gradient-based methods and how different algorithms address them.
• Provide guidance on selecting optimizers and discuss parallel/distributed SGD and additional optimization strategies.

Proposed method

• Classifies gradient descent into batch, stochastic, and mini-batch variants and discusses their trade-offs.
• Derives and presents update rules for common optimizers (Momentum, Nesterov, Adagrad, Adadelta, RMSprop, Adam, AdaMax, Nadam).
• Offers intuition through visualizations and comparisons of update dynamics on loss surfaces and saddle points.
• Reviews parallel/distributed SGD architectures (Hogwild!, Downpour, Delay-tolerant methods, TensorFlow, Elastic Averaging SGD).
• Suggests additional training enhancements (shuffling, curriculum learning, batch normalization, early stopping, gradient noise).

Experimental results

Research questions

• RQ1 What are the main gradient descent variants and how do they differ in data usage and update frequency?
• RQ2 How do popular optimization algorithms mitigate common training challenges (learning rate selection, saddle points, sparse data) in neural networks?
• RQ3 Which optimizers perform best in practice for different data characteristics (e.g., sparse vs dense, non-convex landscapes)?
• RQ4 How can gradient descent be scaled via parallel and distributed architectures without sacrificing convergence?
• RQ5 What auxiliary strategies further improve SGD performance during training?

Key findings

• Mini-batch gradient descent is the most popular variant for neural networks due to a balance between update stability and computational efficiency.
• Adaptive learning-rate methods (Adagrad, Adadelta, RMSprop, Adam, and variants) often outperform vanilla SGD, especially on sparse data or large-scale models, with Adam offering strong empirical performance and bias correction.
• Momentum and Nesterov acceleration can speed up convergence and improve responsiveness, particularly in ravines and near local optima.
• Parallel and distributed SGD approaches (Hogwild!, Downpour, Elastic Averaging) enable faster training on large datasets, with considerations for synchronization and convergence.
• Batch normalization and curriculum learning are valuable auxiliary strategies that can accelerate training and improve generalization.
• In practice, RMSprop, Adadelta, and Adam are highlighted as robust default choices, with Adam often providing the best overall performance among adaptive methods.