[Paper Review] Low-rank Solutions of Linear Matrix Equations via Procrustes Flow

Stephen Tu, Ross Boczar+3 more|arXiv (Cornell University)|Jul 13, 2015

Sparse and Compressive Sensing TechniquesEngineering29 references143 citations

TL;DR

This paper proposes Procrustes Flow, a two-stage algorithm for recovering low-rank matrices from linear measurements using a combination of thresholded gradient initialization and non-convex gradient descent on a low-rank factorization. It establishes geometric convergence to the true matrix under standard restricted isometry conditions, requiring only $O((n_1 + n_2)r)$ Gaussian measurements for exact recovery, matching the information-theoretic lower bound up to a constant factor.

ABSTRACT

In this paper we study the problem of recovering a low-rank matrix from linear measurements. Our algorithm, which we call Procrustes Flow, starts from an initial estimate obtained by a thresholding scheme followed by gradient descent on a non-convex objective. We show that as long as the measurements obey a standard restricted isometry property, our algorithm converges to the unknown matrix at a geometric rate. In the case of Gaussian measurements, such convergence occurs for a $n_1 imes n_2$ matrix of rank $r$ when the number of measurements exceeds a constant times $(n_1+n_2)r$.

Motivation & Objective

• To develop a provably convergent algorithm for low-rank matrix recovery from linear measurements.
• To address the lack of theoretical guarantees in existing non-convex heuristics for matrix sensing.
• To show that gradient descent on a low-rank factorization converges geometrically under standard RIP conditions.
• To establish sample complexity bounds matching the information-theoretic minimum up to a constant factor for Gaussian measurements.

Proposed method

• The algorithm uses a two-phase approach: first, a projected gradient method on the matrix space to obtain an initial low-rank estimate.
• The initialization phase applies power iterations with hard thresholding to maintain rank-$r$ structure, converging linearly to the true matrix.
• The refinement phase performs non-convex gradient descent on the low-rank factors $\bm{U}, \bm{V}$ to minimize the squared misfit $\|\mathcal{A}(\bm{U}\bm{V}^T) - \bm{b}\|_2^2$.
• For the positive semidefinite case, the method optimizes over $\bm{U} \in \mathbb{R}^{n \times r}$ with $\bm{M} = \bm{U}\bm{U}^T$, using a non-convex objective function.
• Theoretical analysis leverages restricted isometry properties (RIP) and establishes that the iterates converge geometrically when $\delta_{4r} \leq 1/25$.
• Key technical tools include distance bounds between low-rank factorizations and matrix perturbation inequalities to control the geometry of the non-convex landscape.

Experimental results

Research questions

• RQ1 Can a non-convex optimization method for low-rank matrix recovery achieve geometric convergence with provable guarantees under standard measurement conditions?
• RQ2 What is the minimal number of linear measurements required for exact recovery using a local search heuristic?
• RQ3 Does a carefully designed initialization via projected gradient descent enable convergence to the global optimum in the low-rank matrix sensing problem?
• RQ4 How does the sample complexity of the Procrustes Flow algorithm scale with matrix dimensions and rank compared to information-theoretic limits?
• RQ5 Under what conditions does gradient descent on the low-rank factorization avoid spurious local minima?

Key findings

• The Procrustes Flow algorithm converges geometrically to the true low-rank matrix when the measurement operator $\mathcal{A}$ satisfies the restricted isometry property with $\delta_{4r} \leq 1/25$.
• For Gaussian measurements, exact recovery is guaranteed with $O((n_1 + n_2)r)$ measurements, matching the number of parameters up to a constant factor.
• The initialization phase converges linearly to a neighborhood of the true matrix, with convergence rate $\|\widetilde{\bm{M}}_\tau - \bm{M}\|_F \leq (2/25)^\tau \|\bm{M}\|_F$.
• The refinement phase achieves geometric convergence to the true solution once the initial estimate is within a constant factor of the true matrix in Frobenius norm.
• The algorithm is robust to noise and maintains stability under small perturbations due to the strong curvature of the non-convex objective near the solution.
• Theoretical analysis confirms that the method avoids spurious local minima and saddle points under standard RIP assumptions, ensuring global convergence from good initialization.