[Paper Review] Practical Accelerated Optimization on Riemannian Manifolds

Foivos Alimisis, Antonio Orvieto+2 more|arXiv (Cornell University)|Feb 11, 2020

Morphological variations and asymmetry8 citations

TL;DR

This paper introduces a new Riemannian descent algorithm that achieves accelerated convergence for geodesically convex and weakly-quasi-convex functions on Riemannian manifolds. By employing a novel estimate sequence, the method establishes convergence rates dependent on manifold curvature, validated empirically on spheres and positive definite matrices.

ABSTRACT

We develop a new Riemannian descent algorithm with an accelerated rate of convergence. We focus on functions that are geodesically convex or weakly-quasi-convex, which are weaker function classes compared to prior work that has considered geodesically strongly convex functions. Our proof of convergence relies on a novel estimate sequence which allows to demonstrate the dependency of the convergence rate on the curvature of the manifold. We validate our theoretical results empirically on several optimization problems defined on a sphere and on the manifold of positive definite matrices.

Motivation & Objective

• To develop a practical Riemannian optimization algorithm with accelerated convergence for broader function classes than prior work.
• To extend convergence guarantees beyond geodesically strongly convex functions to include geodesically convex and weakly-quasi-convex functions.
• To establish a theoretical link between convergence rate and the curvature of the underlying Riemannian manifold.
• To validate the theoretical findings through empirical evaluation on manifold optimization problems involving spheres and positive definite matrices.

Proposed method

• Introduces a novel estimate sequence technique tailored for Riemannian optimization to analyze convergence under weaker convexity assumptions.
• Adapts the estimate sequence framework to account for the curvature of the Riemannian manifold, enabling tighter convergence bounds.
• Derives convergence rates that explicitly depend on the sectional curvature of the manifold, generalizing prior results.
• Applies the algorithm to optimization problems on the sphere and the manifold of positive definite matrices using standard Riemannian optimization tools.
• Employs retraction and vector transport operations to ensure numerical stability and manifold consistency during iterations.
• Uses the estimate sequence to bound the decrease in objective function value per iteration, leading to accelerated convergence.

Experimental results

Research questions

• RQ1 Can accelerated convergence be achieved on Riemannian manifolds for functions that are geodesically convex or weakly-quasi-convex, rather than just strongly convex?
• RQ2 How does the curvature of the Riemannian manifold influence the convergence rate of the proposed algorithm?
• RQ3 Can a novel estimate sequence framework be designed to handle the geometric structure of manifolds while ensuring acceleration?
• RQ4 Does the proposed method outperform standard Riemannian descent in practice on non-Euclidean domains?
• RQ5 What is the empirical performance of the algorithm on standard manifolds such as the sphere and the manifold of positive definite matrices?

Key findings

• The proposed algorithm achieves an accelerated convergence rate for geodesically convex and weakly-quasi-convex functions on Riemannian manifolds.
• The convergence rate is explicitly dependent on the sectional curvature of the manifold, with tighter bounds in non-positive curvature regions.
• The novel estimate sequence technique successfully generalizes to non-strongly convex settings on manifolds, enabling accelerated convergence.
• Empirical results confirm the theoretical convergence rates on the sphere and the manifold of positive definite matrices.
• The algorithm demonstrates faster convergence compared to standard Riemannian gradient descent in tested optimization problems.
• The method maintains stability and efficiency across different manifold geometries, including those with non-trivial curvature.