[Paper Review] Riemannian Perspective on Matrix Factorization

Kwangjun Ahn, Felipe Suárez|arXiv (Cornell University)|Feb 1, 2021

Sparse and Compressive Sensing Techniques42 references4 citations

TL;DR

This paper presents a Riemannian geometric framework for matrix factorization in matrix completion, analyzing the optimization landscape via Grassmannian manifolds and principal angles. It establishes that the cost function is geodesically convex within a specific region around the true subspace, and all critical points outside this region are strictly saddle points, providing a theoretical foundation for convergence of Riemannian optimization methods.

ABSTRACT

We study the non-convex matrix factorization approach to matrix completion via Riemannian geometry. Based on an optimization formulation over a Grassmannian manifold, we characterize the landscape based on the notion of principal angles between subspaces. For the fully observed case, our results show that there is a region in which the cost is geodesically convex, and outside of which all critical points are strictly saddle. We empirically study the partially observed case based on our findings.

Motivation & Objective

• To understand the non-convex optimization landscape of matrix factorization in matrix completion using Riemannian geometry.
• To characterize the geometry of the cost function on the Grassmann manifold using principal angles between subspaces.
• To identify a region where the cost function is geodesically convex and where all critical points are strictly saddle points.
• To extend insights from the fully observed case to the partially observed case through empirical validation.
• To provide theoretical justification for the effectiveness of Riemannian optimization in matrix completion.

Proposed method

• Formulates matrix factorization as an optimization over a single Grassmann manifold, parameterizing low-rank matrices via their column subspaces.
• Uses principal angles between subspaces to characterize the geometric structure of the cost function's landscape.
• Defines a neighborhood $\mathcal{N}_{[\mathbf{U}]}(\pi/4)$ around the true subspace where the cost function is geodesically convex.
• Employs Riemannian optimization tools, including geodesic convexity and second derivative analysis, to analyze critical points.
• Compares the partially observed case to the fully observed case by varying sampling probabilities $p$ and observing landscape similarity.
• Uses numerical experiments to validate that the landscape in the partially observed case closely resembles the fully observed case for moderate $p$.

Experimental results

Research questions

• RQ1 Under what geometric conditions is the matrix factorization cost function geodesically convex on the Grassmann manifold?
• RQ2 Can the landscape of matrix factorization be characterized using principal angles between subspaces?
• RQ3 Are all critical points outside the convex region strictly saddle points?
• RQ4 How does the optimization landscape in the partially observed case compare to the fully observed case?
• RQ5 Can the theoretical insights from the fully observed case be extended to the partially observed setting?

Key findings

• The cost function is geodesically convex within a neighborhood $\mathcal{N}_{[\mathbf{U}]}(\pi/4)$ of the true subspace, defined via principal angles.
• All critical points outside this region are strictly saddle points, as confirmed by negative second derivatives along escaping directions.
• Empirical results show that the landscape for the partially observed case closely resembles the fully observed case when the sampling probability $p$ is not too small (e.g., $p \geq 0.01$).
• The cost function can become discontinuous in the partially observed case under standard formulations, as demonstrated by a counterexample with a discontinuity at $\mathbf{x} = [1, 0, 0]^\top$.
• Alternative formulations using chordal distance or regularization can restore continuity and improve numerical stability.
• Riemannian gradient descent trajectories remain stable and converge within the convex region, suggesting favorable optimization behavior.