[Paper Review] Riemannian SVRG: Fast Stochastic Optimization on Riemannian Manifolds

Hongyi Zhang, Sashank J. Reddi+1 more|arXiv (Cornell University)|May 23, 2016

3D Shape Modeling and Analysis36 references85 citations

TL;DR

This paper introduces Riemannian SVRG, the first variance-reduced stochastic optimization method for finite-sum problems on Riemannian manifolds. It achieves global linear convergence for geodesically strongly convex functions and provides the first non-asymptotic complexity analysis for nonconvex Riemannian optimization, with convergence rates dependent on manifold curvature.

ABSTRACT

We study optimization of finite sums of geodesically smooth functions on Riemannian manifolds. Although variance reduction techniques for optimizing finite-sums have witnessed tremendous attention in the recent years, existing work is limited to vector space problems. We introduce Riemannian SVRG (RSVRG), a new variance reduced Riemannian optimization method. We analyze RSVRG for both geodesically convex and nonconvex (smooth) functions. Our analysis reveals that RSVRG inherits advantages of the usual SVRG method, but with factors depending on curvature of the manifold that influence its convergence. To our knowledge, RSVRG is the first provably fast stochastic Riemannian method. Moreover, our paper presents the first non-asymptotic complexity analysis (novel even for the batch setting) for nonconvex Riemannian optimization. Our results have several implications; for instance, they offer a Riemannian perspective on variance reduced PCA, which promises a short, transparent convergence analysis.

Motivation & Objective

• Address the lack of fast stochastic optimization methods for Riemannian finite-sum problems, which are common in machine learning and matrix geometry.
• Overcome the limitations of standard Riemannian stochastic and full gradient methods, which suffer from slow convergence or high per-iteration cost.
• Develop a variance-reduced framework tailored to Riemannian geometry, enabling faster convergence for large-scale problems on curved manifolds.
• Provide the first non-asymptotic global convergence analysis for stochastic Riemannian optimization in both convex and nonconvex settings.
• Demonstrate the method's effectiveness through applications to leading eigenvector computation and Riemannian centroid estimation on positive definite matrices.

Proposed method

• Propose Riemannian SVRG (Rsvrg), a stochastic variance reduction method adapted to Riemannian manifolds using the SVRG framework.
• Leverage the Riemannian exponential map and retraction to define updates on the manifold, avoiding projection steps.
• Introduce a novel theoretical analysis that accounts for manifold curvature, particularly sectional curvature, in convergence rate bounds.
• Use geodesic smoothness and geodesic convexity assumptions to derive convergence guarantees under both convex and nonconvex settings.
• Apply the method to specific problems such as variance-reduced PCA and Riemannian centroid computation, showing improved convergence over baseline methods.
• Establish convergence rates that depend on curvature parameters, with optimal stepsize scaling as $ O(1/( heta^3 n)) $, where $ heta $ relates to manifold curvature.

Experimental results

Research questions

• RQ1 Can variance reduction techniques from Euclidean space be successfully extended to Riemannian finite-sum optimization problems?
• RQ2 What are the global non-asymptotic convergence rates of stochastic Riemannian optimization for geodesically nonconvex and non-gradient-dominated functions?
• RQ3 How does manifold curvature influence the convergence behavior of stochastic Riemannian optimization algorithms?
• RQ4 Can Riemannian SVRG achieve linear convergence for geodesically strongly convex problems on manifolds, and how does it compare to batch and stochastic gradient methods?
• RQ5 What insights does the Riemannian formulation provide for classical problems like PCA or Riemannian centroid computation?

Key findings

• Riemannian SVRG achieves global linear convergence for geodesically strongly convex functions, with convergence rates explicitly dependent on manifold curvature.
• The method provides the first non-asymptotic complexity analysis for stochastic Riemannian optimization in the nonconvex setting, even in the batch case.
• For variance-reduced PCA, the Riemannian formulation offers a transparent and short convergence analysis, explaining the fast convergence of VR-PCA.
• Empirical results show Riemannian SVRG outperforms Riemannian full gradient (RGD) and stochastic gradient (RSGD) methods in terms of iteration complexity, especially for large-scale problems.
• In the Riemannian centroid computation task, Rsvrg achieves linear convergence and significantly reduces the number of oracle calls compared to baseline methods.
• The IFO complexity of Rsvrg scales linearly with $ 1/ ext{eigengap} $, matching the performance of VR-PCA and confirming a close geometric connection between the two methods.