[Paper Review] Gradient Descent based Optimization Algorithms for Deep Learning Models Training

Jiawei Zhang|arXiv (Cornell University)|Mar 11, 2019

Stochastic Gradient Optimization Techniques10 references45 citations

TL;DR

A comprehensive introduction to gradient-descent–based optimization methods for training deep neural networks, covering vanilla, stochastic, mini-batch, momentum, Nesterov, adaptive methods (Adagrad, RMSprop, Adadelta), and hybrid approaches like Gadam, with formal definitions, equations, and algorithm pseudo-codes.

ABSTRACT

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back propagation algorithm actually. In back propagation, the model variables will be updated iteratively until convergence with gradient descent based optimization algorithms. Besides the conventional vanilla gradient descent algorithm, many gradient descent variants have also been proposed in recent years to improve the learning performance, including Momentum, Adagrad, Adam, Gadam, etc., which will all be introduced in this paper respectively.

Motivation & Objective

• Introduce gradient descent–based optimization for training deep neural networks.
• Present and compare conventional, momentum, and adaptive gradient methods.
• Provide formal equations and pseudo-code for core optimization algorithms.
• Discuss practical considerations such as learning rate, convergence, and saddle points.

Proposed method

• Define the learning objective for deep networks as minimizing L(theta; T).
• Present vanilla gradient descent, SGD, and mini-batch GD with their update rules.
• Introduce Momentum and Nesterov Accelerated Gradient with their look-ahead/update schemes.
• Describe adaptive gradient methods (Adagrad, RMSprop, Adadelta) and their per-parameter learning rates.
• Provide algorithmic pseudo-code for each method and discuss convergence and stability considerations.

Experimental results

Research questions

• RQ1 What are the fundamental gradient-descent–based algorithms for training deep learning models, and how do they differ in data usage and update rules?
• RQ2 How do momentum and look-ahead methods (Nesterov) improve convergence over standard gradient descent?
• RQ3 How do adaptive gradient methods adjust learning rates across parameters and over time, and what are their trade-offs?
• RQ4 What are the practical considerations (learning rate, convergence criteria, saddle points) that affect training stability and performance of these algorithms?
• RQ5 How can hybrid approaches (e.g., Gadam) combine gradient-based learning with evolutionary strategies to optimize model training?

Key findings

• Provides formal definitions of vanilla GD, SGD, and mini-batch GD and their update equations.
• Derives and explains Momentum and Nesterov Accelerated Gradient and their impact on convergence behavior.
• Details adaptive gradient methods Adagrad, RMSprop, and Adadelta with per-parameter learning rates and decay mechanisms.
• Offers algorithmic pseudo-code for major methods, enabling practical implementation and comparison.
• Discusses practical challenges such as learning rate selection, adjustment, and saddle-point avoidance in training deep nets.
• Introduces hybrid evolutionary gradient concepts (Gadam) as a variant integrating gradient descent with genetic evolution.