[Paper Review] Extending FISTA to Riemannian Optimization for Sparse PCA

Wen Huang, Kexiang Wei|arXiv (Cornell University)|Sep 12, 2019

Matrix Theory and Algorithms1 references9 citations

TL;DR

This paper extends the FISTA algorithm to Riemannian optimization for sparse PCA, proposing an accelerated Riemannian proximal gradient method on the Stiefel manifold to simultaneously enforce sparsity and orthogonality. The method achieves faster convergence and improved efficiency through a safeguard strategy and diagonal weighting, with global convergence to stationary points rigorously proven.

ABSTRACT

Sparse PCA, an important variant of PCA, attempts to find sparse loading vectors when conducting dimension reduction. This paper considers the Riemannian optimization problem related to the ScoTLASS model for the sparse PCA which can impose orthogonality and sparsity simultaneously. We extend FISTA from the Euclidean space to the Riemannian manifold to solve this problem, leading to the accelerated Riemannian proximal gradient method. Since the optimization problem is essentially non-convex, a safeguard strategy is introduced in the algorithm. Moreover, a diagonal weighting strategy is also proposed which can further improve the computational efficiency of the Riemannian proximal methods. Numerical evaluations establish the computational advantages of the proposed methods over the existing proximal gradient methods on manifold. Convergence of the methods to stationary point has also been rigorously justified.

Motivation & Objective

• To address the challenge of simultaneously enforcing sparsity and orthogonality in sparse PCA using Riemannian optimization.
• To extend the FISTA algorithm, originally designed for Euclidean spaces, to non-convex Riemannian manifolds.
• To improve computational efficiency of proximal gradient methods on manifolds through a novel diagonal weighting strategy.
• To ensure convergence to stationary points despite the non-convex nature of the optimization problem.
• To provide a robust and efficient algorithm for solving the ScoTLASS model in sparse PCA.

Proposed method

• The FISTA algorithm is generalized to Riemannian manifolds by incorporating Riemannian geometry, specifically on the Stiefel manifold, to handle orthogonality constraints.
• An accelerated Riemannian proximal gradient method is developed, combining momentum steps with Riemannian retractions and vector transport.
• A safeguard strategy is introduced to ensure sufficient decrease in the objective function, addressing non-convexity and preventing stagnation.
• A diagonal weighting strategy is proposed to adaptively scale the proximal term, enhancing convergence speed and numerical stability.
• The method leverages the Riemannian structure to maintain orthogonality of loading vectors throughout the optimization process.
• Convergence to a stationary point is rigorously established under standard assumptions for non-convex Riemannian optimization.

Experimental results

Research questions

• RQ1 Can FISTA be effectively extended to Riemannian optimization for non-convex sparse PCA problems?
• RQ2 How can acceleration and globalization strategies be adapted to Riemannian proximal methods for improved convergence?
• RQ3 What impact does diagonal weighting have on the computational efficiency of Riemannian proximal algorithms?
• RQ4 Does the proposed method achieve faster convergence and better performance compared to existing manifold-based proximal methods?
• RQ5 Is convergence to a stationary point guaranteed under the proposed algorithmic framework?

Key findings

• The proposed accelerated Riemannian proximal gradient method achieves faster convergence than existing proximal gradient methods on the Stiefel manifold.
• The safeguard strategy effectively prevents convergence issues in the non-convex setting, ensuring robust progress.
• The diagonal weighting strategy significantly improves computational efficiency by adapting to the local geometry of the manifold.
• Theoretical convergence to a stationary point is rigorously established, validating the algorithm's reliability.
• Numerical experiments demonstrate consistent computational advantages across various sparse PCA test cases.
• The method successfully enforces both sparsity and orthogonality in loading vectors, as required by the ScoTLASS model.