[Paper Review] Riemannian stochastic variance reduced gradient

Hiroyuki Sato, Hiroyuki Kasai+1 more|arXiv (Cornell University)|Feb 18, 2017

Topological and Geometric Data Analysis19 citations

TL;DR

This paper proposes R-SVRG, a Riemannian stochastic variance reduced gradient algorithm that extends SVRG to manifold-structured optimization by leveraging retraction and vector transport to handle gradient operations on non-Euclidean spaces. It achieves global convergence with decaying step sizes and local superlinear convergence with fixed steps, outperforming Riemannian stochastic gradient descent on SPD manifolds and Grassmann manifolds in centroid computation, PCA, and low-rank matrix completion.

ABSTRACT

In recent years, stochastic variance reduction algorithms have attracted considerable attention for minimizing the average of a large but finite number of loss functions. This paper proposes a novel Riemannian extension of the Euclidean stochastic variance reduced gradient (R-SVRG) algorithm to a manifold search space. The key challenges of averaging, adding, and subtracting multiple gradients are addressed with retraction and vector transport. For the proposed algorithm, we present a global convergence analysis with a decaying step size as well as a local convergence rate analysis with a fixed step size under some natural assumptions. In addition, the proposed algorithm is applied to the computation problem of the Riemannian centroid on the symmetric positive definite (SPD) manifold as well as the principal component analysis and low-rank matrix completion problems on the Grassmann manifold. The results show that the proposed algorithm outperforms the standard Riemannian stochastic gradient descent algorithm in each case.

Motivation & Objective

• To address the challenge of applying variance reduction techniques to optimization on Riemannian manifolds where standard Euclidean operations like averaging and vector addition are inapplicable.
• To develop a stochastic optimization algorithm that maintains the convergence advantages of SVRG in Euclidean spaces while adapting to the geometric structure of manifolds.
• To establish global and local convergence guarantees for the proposed algorithm under natural assumptions on the manifold and objective function.
• To demonstrate the practical superiority of R-SVRG over standard Riemannian stochastic gradient descent in real-world problems on symmetric positive definite and Grassmann manifolds.

Proposed method

• The algorithm extends the Euclidean SVRG framework to Riemannian manifolds by replacing Euclidean operations with Riemannian counterparts: retraction for approximate exponential mapping and vector transport for parallel transport of gradients.
• Gradient differences are computed and updated using Riemannian operations to maintain variance reduction without requiring full gradient computations at each iteration.
• A decaying step size ensures global convergence to a critical point under standard assumptions on the objective function and manifold geometry.
• A fixed step size yields local linear convergence under additional regularity conditions, such as Lipschitz continuous Hessian on the manifold.
• The method is instantiated on two manifolds: the symmetric positive definite (SPD) manifold for centroid computation and the Grassmann manifold for PCA and low-rank matrix completion.
• The algorithm uses manifold-specific retraction and vector transport operators tailored to the geometry of SPD matrices and Grassmannian subspaces.

Experimental results

Research questions

• RQ1 Can variance reduction techniques from Euclidean optimization be effectively extended to Riemannian manifolds where standard vector operations are not defined?
• RQ2 How can retraction and vector transport be used to replace averaging and subtraction of gradients in SVRG on manifolds?
• RQ3 What convergence guarantees (global and local) can be established for the proposed Riemannian SVRG algorithm under natural geometric assumptions?
• RQ4 Does R-SVRG outperform standard Riemannian stochastic gradient descent in practical manifold learning tasks?

Key findings

• R-SVRG achieves global convergence with a decaying step size under standard assumptions, ensuring convergence to a critical point on the manifold.
• With a fixed step size, the algorithm exhibits local linear convergence under additional regularity conditions, indicating faster convergence near optimal solutions.
• On the symmetric positive definite manifold, R-SVRG significantly outperforms Riemannian stochastic gradient descent in computing the Riemannian centroid.
• In principal component analysis on the Grassmann manifold, R-SVRG converges faster and with greater stability than RSGD.
• For low-rank matrix completion on the Grassmann manifold, R-SVRG demonstrates superior convergence speed and accuracy compared to RSGD.
• The use of retraction and vector transport enables effective variance reduction without requiring full gradient evaluations, reducing computational overhead.