[Paper Review] Simple algorithms for optimization on Riemannian manifolds with constraints

Changshuo Liu, Nicolas Boumal|arXiv (Cornell University)|Jan 28, 2019

Advanced Optimization Algorithms Research42 references4 citations

TL;DR

This paper extends classical constrained optimization methods—augmented Lagrangian and exact penalty methods—to Riemannian manifolds with constraints, preserving geometric structure for improved performance. It demonstrates that exploiting Riemannian geometry yields superior computational efficiency and accuracy in high-dimensional problems like non-negative PCA, k-means, and minimum balanced cut, especially when compared to treating the manifold as a set of equality constraints in Euclidean space.

ABSTRACT

We consider optimization problems on manifolds with equality and inequality constraints. A large body of work treats constrained optimization in Euclidean spaces. In this work, we consider extensions of existing algorithms from the Euclidean case to the Riemannian case. Thus, the variable lives on a known smooth manifold and is further constrained. In doing so, we exploit the growing literature on unconstrained Riemannian optimization. For the special case where the manifold is itself described by equality constraints, one could in principle treat the whole problem as a constrained problem in a Euclidean space. The main hypothesis we test here is whether it is sometimes better to exploit the geometry of the constraints, even if only for a subset of them. Specifically, this paper extends an augmented Lagrangian method and smoothed versions of an exact penalty method to the Riemannian case, together with some fundamental convergence results. Numerical experiments indicate some gains in computational efficiency and accuracy in some regimes for minimum balanced cut, non-negative PCA and $k$-means, especially in high dimensions.

Motivation & Objective

• To systematically extend classical constrained optimization algorithms—augmented Lagrangian and exact penalty methods—from Euclidean spaces to Riemannian manifolds with equality and inequality constraints.
• To investigate whether exploiting the intrinsic geometry of the manifold constraint improves convergence and computational efficiency compared to treating the manifold as additional equality constraints in a Euclidean setting.
• To establish fundamental convergence properties (local and global) for the Riemannian extensions of these methods.
• To evaluate the performance of the proposed Riemannian algorithms on real-world problems such as non-negative PCA, k-means, and minimum balanced cut.
• To provide a comparative benchmark against standard nonlinear programming solvers like fmincon, which do not exploit Riemannian structure.

Proposed method

• Adapt the augmented Lagrangian method (ALM) to Riemannian manifolds by incorporating manifold-specific retractions and vector transport, ensuring iterates remain on the manifold.
• Extend the exact penalty method to Riemannian settings by formulating a mini-sum-max subproblem involving maximum functions of constraint violations, which is nonsmooth and requires specialized solution techniques.
• Develop a robust subgradient descent method without gradient sampling for solving the mini-sum-max subproblems arising in the exact penalty method.
• Apply smoothing techniques using two classical smoothing functions—log-sum-exp and 'linear-quadratic + Huber loss'—to approximate the nonsmooth penalty subproblems.
• Integrate the Riemannian optimization framework with existing algorithms, maintaining the manifold constraint to numerical accuracy at every iteration.
• Implement and compare the Riemannian algorithms against a baseline fmincon solver that treats the manifold as a set of equality constraints in Euclidean space.

Experimental results

Research questions

• RQ1 Can classical constrained optimization methods like augmented Lagrangian and exact penalty methods be successfully extended to Riemannian manifolds with both equality and inequality constraints?
• RQ2 Does exploiting the Riemannian geometry of the manifold constraint lead to improved convergence speed and accuracy compared to treating the manifold as a set of equality constraints in Euclidean space?
• RQ3 How do the convergence properties of the Riemannian extensions of these methods compare to their Euclidean counterparts?
• RQ4 What is the empirical performance of Riemannian ALM and exact penalty methods on high-dimensional problems such as non-negative PCA, k-means, and minimum balanced cut?
• RQ5 Are there specific problem regimes (e.g., high dimensionality) where Riemannian methods significantly outperform standard nonlinear programming solvers like fmincon?

Key findings

• In high-dimensional problems, Riemannian algorithms that exploit the manifold structure outperform the baseline fmincon solver, which treats the manifold as additional equality constraints.
• For non-negative PCA, the Riemannian exact penalty method achieved a maximum constraint violation of $2.13 \cdot 10^{-7}$, significantly lower than fmincon’s $4.44 \cdot 10^{-5}$.
• In k-means clustering, the Riemannian augmented Lagrangian method reduced computation time to $1.70 \cdot 10^{2}$ seconds on the Wine dataset, compared to $3.86 \cdot 10^{3}$ seconds for fmincon.
• On the Pima dataset, the Riemannian exact penalty method achieved a cost of $9.48 \cdot 10^{-3}$, outperforming fmincon’s $2.60 \cdot 10^{-2}$, with a maximum violation of $2.10 \cdot 10^{-10}$.
• For the Cloud dataset (k=4), the Riemannian exact penalty method achieved the lowest cost ($5.49 \cdot 10^{-4}$) and fastest time ($3.65 \cdot 10^{3}$ seconds), outperforming other methods.
• The proposed robust subgradient method for mini-sum-max problems showed improved stability and performance without requiring gradient sampling, suggesting broader applicability in nonsmooth Riemannian optimization.