[Paper Review] Unconstrained optimisation on Riemannian manifolds

Tuyen Trung Truong|arXiv (Cornell University)|Aug 25, 2020

Stochastic Gradient Optimization Techniques23 references4 citations

TL;DR

This paper extends Backtracking Gradient Descent and New Q-Newton’s method to Riemannian manifolds using Nash’s embedding theorem and local retractions, proving almost-sure convergence to local minima or divergence to infinity for random hyperparameters. For $C^3$ Morse functions with compact sublevels, convergence to non-degenerate minima occurs with quadratic rate, and saddle points are globally avoided under specific group structure assumptions.

ABSTRACT

In this paper, we give explicit descriptions of versions of (Local-) Backtracking Gradient Descent and New Q-Newton's method to the Riemannian setting.Here are some easy to state consequences of results in this paper, where X is a general Riemannian manifold of finite dimension and $f:X ightarrow \mathbb{R}$ a $C^2$ function which is Morse (that is, all its critical points are non-degenerate). {\bf Theorem.} For random choices of the hyperparameters in the Riemanian Local Backtracking Gradient Descent algorithm and for random choices of the initial point $x_0$, the sequence $\{x_n\}$ constructed by the algorithm either (i) converges to a local minimum of $f$ or (ii) eventually leaves every compact subsets of $X$ (in other words, diverges to infinity on $X$). If $f$ has compact sublevels, then only the former alternative happens. The convergence rate is the same as in the classical paper by Armijo. {\bf Theorem.} Assume that $f$ is $C^3$. For random choices of the hyperparametes in the Riemannian New Q-Newton's method, if the sequence constructed by the algorithm converges, then the limit is a critical point of $f$. We have a local Stable-Center manifold theorem, near saddle points of $f$, for the dynamical system associated to the algorithm. If the limit point is a non-degenerate minimum point, then the rate of convergence is quadratic. If moreover $X$ is an open subset of a Lie group and the initial point $x_0$ is chosen randomly, then we can globally avoid saddle points. As an application, we propose a general method using Riemannian Backtracking GD to find minimum of a function on a bounded ball in a Euclidean space, and do explicit calculations for calculating the smallest eigenvalue of a symmetric square matrix.

Motivation & Objective

• To generalize convergence and saddle-point avoidance results of Backtracking GD and New Q-Newton’s method from Euclidean and Banach spaces to general finite-dimensional Riemannian manifolds.
• To establish theoretical guarantees for iterative optimisation on manifolds using only local geometric and function information.
• To demonstrate practical benefits of Riemannian optimisation for constrained problems and singular cost functions, including eigenvalue computation.
• To provide a general framework for solving constrained optimisation on compact Riemannian submanifolds, such as the sphere, via Riemannian methods.

Proposed method

• Adapts Local Backtracking Gradient Descent and New Q-Newton’s method to Riemannian manifolds using Riemannian retractions and vector transport.
• Applies Nash’s embedding theorem to isometrically embed a Riemannian manifold into Euclidean space, enabling transfer of convergence results from Euclidean settings.
• Employs good local retractions to ensure local consistency and stability of the optimisation dynamics on the manifold.
• Uses random hyperparameters and initial points to achieve almost-sure convergence properties under mild regularity conditions.
• Introduces a Riemannian version of the New Q-Newton’s method with a stable-center manifold theorem near saddle points.
• Applies the framework to constrained problems like minimizing a function on the unit ball, with explicit computation for quadratic forms and eigenvalue problems.

Experimental results

Research questions

• RQ1 Can the convergence and saddle-point avoidance properties of Backtracking GD and New Q-Newton’s method be extended to Riemannian manifolds?
• RQ2 Can global convergence to non-degenerate minima and avoidance of saddle points be guaranteed under random hyperparameters and initial conditions on general Riemannian manifolds?
• RQ3 Can the Riemannian New Q-Newton’s method achieve quadratic convergence near non-degenerate minima and avoid saddle points when the manifold is an open Lie group?
• RQ4 Can Riemannian optimisation outperform standard Euclidean methods for constrained or singular optimisation problems?
• RQ5 Is there a general, implementable method for constrained optimisation on compact Riemannian manifolds using Riemannian Backtracking GD?

Key findings

• For random hyperparameters and initial points, Riemannian Local Backtracking GD converges to a local minimum or diverges to infinity; if sublevels are compact, only convergence to a local minimum occurs.
• The convergence rate of Riemannian Local Backtracking GD matches Armijo’s classical rate in the Euclidean setting.
• For $C^3$ Morse functions, Riemannian New Q-Newton’s method converges to a critical point if the sequence converges, with quadratic convergence rate at non-degenerate minima.
• A local stable-center manifold theorem is established near saddle points for the Riemannian New Q-Newton’s method, enabling theoretical analysis of dynamics.
• When the manifold is an open subset of a Lie group and the initial point is random, the Riemannian New Q-Newton’s method globally avoids saddle points.
• Experiments show Riemannian optimisation outperforms standard GD on constrained problems and singular functions, e.g., computing the smallest eigenvalue of a symmetric matrix via minimisation on the sphere.