[Paper Review] Asymptotic behaviour of learning rates in Armijo's condition

Tuyen Trung Truong, Tuan Hang Nguyen|arXiv (Cornell University)|Jul 7, 2020

Stochastic Gradient Optimization Techniques5 references5 citations

TL;DR

This paper analyzes the asymptotic behavior of learning rates in Armijo’s condition within Backtracking Gradient Descent. It proves that if a sequence converges to a non-degenerate critical point, the learning rates must be bounded, with explicit quantitative bounds in terms of the Hessian norm and its inverse at the limit point. This shows Unbounded Backtracking GD behaves similarly to standard Backtracking GD near non-degenerate minima, but diverges significantly near degenerate ones.

ABSTRACT

Fix a constant $00$, we say that Armijo's condition is satisfied if $f(x-δ abla f(x))-f(x)\leq -αδ|| abla f(x)||^2$. It is a basis for the well known Backtracking Gradient Descent (Backtracking GD) algorithm. Consider a sequence $\{x_n\}$ defined by $x_{n+1}=x_n-δ_n abla f(x_n)$, for positive numbers $δ_n$ for which Armijo's condition is satisfied. We show that if $\{x_n\}$ converges to a non-degenerate critical point, then $\{δ_n\}$ must be bounded. Moreover this boundedness can be quantified in terms of the norms of the Hessian $ abla ^2f$ and its inverse at the limit point. This complements the first author's results on Unbounded Backtracking GD, and shows that in case of convergence to a non-degenerate critical point the behaviour of Unbounded Backtracking GD is not too different from that of usual Backtracking GD. On the other hand, in case of convergence to a degenerate critical point the behaviours can be very much different. We run some experiments to illustrate that both scenrios can really happen. In another part of the paper, we argue that Backtracking GD has the correct unit (according to a definition by Zeiler in his Adadelta's paper). The main point is that since learning rate in Backtracking GD is bound by Armijo's condition, it is not unitless.

Motivation & Objective

• To investigate whether learning rates in Unbounded Backtracking GD can grow to infinity while still converging to a non-degenerate critical point.
• To compare the asymptotic behavior of learning rates in Unbounded Backtracking GD versus standard Backtracking GD near non-degenerate and degenerate critical points.
• To establish unit correctness of Backtracking GD based on physical dimension analysis, addressing a key limitation of standard and adaptive methods.
• To provide theoretical justification for the convergence behavior of Backtracking GD under Armijo’s condition, especially in relation to Hessian properties at limit points.

Proposed method

• Uses Taylor expansion of $f$ and $\nabla f$ around a non-degenerate critical point $x_\infty$ to derive asymptotic estimates of $f(x_n) - f(x_\infty)$ and $\|\nabla f(x_n)\|$.
• Applies Armijo’s condition $f(x_n - \delta_n \nabla f(x_n)) - f(x_n) \leq -\alpha \delta_n \|\nabla f(x_n)\|^2$ to derive an upper bound on $\delta_n$.
• Employs operator norm estimates for the Hessian $\nabla^2 f(x_\infty)$ and its inverse to quantify the upper bound on $\delta_n$.
• Performs unit analysis using dimensional consistency: shows $\delta(x_n)$ has unit $\text{Unit}(x)^2 / \text{Unit}(f)$, and Armijo’s condition is unit-correct only if $\alpha$ is unitless.
• Compares Backtracking GD with standard GD, Momentum, Adagrad, Newton’s method, and Diminishing GD in terms of unit correctness.
• Conducts numerical experiments to illustrate that both bounded and unbounded learning rate behaviors can occur in practice, depending on the nature of the critical point.

Experimental results

Research questions

• RQ1 Can learning rates in Unbounded Backtracking GD grow to infinity while the sequence converges to a non-degenerate critical point?
• RQ2 What is the quantitative upper bound on learning rates $\delta_n$ in terms of the Hessian and its inverse at the limit point?
• RQ3 How does the behavior of Unbounded Backtracking GD differ from standard Backtracking GD when converging to degenerate versus non-degenerate critical points?
• RQ4 Is Backtracking GD unit-correct according to Zeiler’s unit analysis, and how does it compare to other optimization methods?
• RQ5 Can the learning rate in Backtracking GD be chosen independently of function scale, and what are the implications for convergence and stability?

Key findings

• If a sequence $\{x_n\}$ defined by $x_{n+1} = x_n - \delta_n \nabla f(x_n)$ satisfies Armijo’s condition and converges to a non-degenerate critical point, then $\delta_n$ must be bounded.
• The bound on $\delta_n$ is quantitatively controlled by $\|\nabla^2 f(x_\infty)\|$ and $\|\nabla^2 f(x_\infty)^{-1}\|$, with $\alpha \delta_n \leq \frac{1}{2}(\|\nabla^2 f(x_\infty)\| + \epsilon)(\|\nabla^2 f(x_\infty)^{-1}\| + \epsilon)^2$ for large $n$.
• In contrast, when converging to a degenerate critical point, learning rates in Unbounded Backtracking GD can grow without bound, indicating a fundamental difference in behavior.
• Backtracking GD has correct units: the learning rate $\delta(x_n)$ has unit $\text{Unit}(x)^2 / \text{Unit}(f)$, and Armijo’s condition is unit-correct only if $\alpha$ is unitless.
• Standard GD and Diminishing GD do not have correct units, while Newton’s method does, but is not guaranteed to be a descent method.
• The choice of learning rate $\delta_0 = 1/L$ in standard GD (with $L$ the Lipschitz constant of $\nabla f$) yields unit correctness, and such a method is a special case of Backtracking GD.