[Paper Review] An improvement of the convergence proof of the ADAM-Optimizer

Sebastian Bock, Josef Goppold+1 more|arXiv (Cornell University)|Apr 27, 2018

Stochastic Gradient Optimization Techniques3 references125 citations

TL;DR

The paper offers an improved convergence proof for the ADAM optimizer, addressing gaps in the original Kingma–Ba proof and presenting a corollary showing 1/sqrt(T) convergence under stated conditions, while noting a remaining conjecture.

ABSTRACT

A common way to train neural networks is the Backpropagation. This algorithm includes a gradient descent method, which needs an adaptive step size. In the area of neural networks, the ADAM-Optimizer is one of the most popular adaptive step size methods. It was invented in \cite{Kingma.2015} by Kingma and Ba. The $5865$ citations in only three years shows additionally the importance of the given paper. We discovered that the given convergence proof of the optimizer contains some mistakes, so that the proof will be wrong. In this paper we give an improvement to the convergence proof of the ADAM-Optimizer.

Motivation & Objective

• Motivate the use of adaptive step sizes in neural network training and examine the ADAM optimizer's theoretical convergence.
• Identify and address mistakes in the original convergence proof of ADAM by Kingma and Ba.
• Propose improvements to the convergence analysis and articulate a conjecture that could complete the proof.

Proposed method

• Review the ADAM algorithm and its moment estimates (m_t and v_t) and their updates.
• Introduce a conjecture (Conjecture 4.2) on a bound involving the squared biased moments and past gradients.
• Define the error sum R(T) to quantify the cumulative deviation from the optimum and derive an upper bound for R(T) (Theorem 4.4).
• Deduce a corollary (Corollary 4.5) showing R(T)/T = O(1/√T) under bounded gradients and bounded parameter updates.
• Provide a structured, component-wise (per-coordinate) analysis of the update terms and their contributions to the convergence bound.

Experimental results

Research questions

• RQ1 Does the ADAM optimizer converge for convex and differentiable error functions under the given stochastic update rules?
• RQ2 What is the bound on the cumulative error R(T) and how does it scale with T under the ADAM updates?
• RQ3 Can a corollary be established to show a 1/√T convergence rate for R(T)/T under bounded gradients and bounded parameter differences?
• RQ4 What conjecture or missing piece is required to complete the original convergence proof for ADAM?

Key findings

• An upper bound for the error sum R(T) is established, combining terms involving gradients, step sizes, and moment estimates.
• Under the stated assumptions, R(T) is bounded by terms that scale with √T and with 1/(1−λ)^2, among other factors.
• Corollary 4.5 shows that R(T)/T = O(1/√T), implying a sublinear convergence rate for the ADAM method under bounded gradients and bounded weight updates.
• The analysis relies on a conjecture (Conjecture 4.2) about a gradient-related bound, which is acknowledged as a remaining open piece for completing the original proof.
• The paper concludes with an improved convergence proof and highlights that fully solidifying the conjecture could strengthen confidence in ADAM’s convergence properties.