[Paper Review] Run Procrustes, Run! On the convergence of accelerated Procrustes Flow.

Anastasios Kyrillidis, Shashanka Ubaru+2 more|arXiv (Cornell University)|Jun 1, 2018

Stochastic Gradient Optimization Techniques57 references3 citations

TL;DR

This paper establishes linear convergence of accelerated Procrustes Flow for low-rank matrix sensing under non-convex optimization, showing that acceleration achieves the same condition-number dependence as non-accelerated methods while improving practical performance on synthetic and real-world tasks like neuronal activity recovery and quantum state tomography.

ABSTRACT

In this work, we present theoretical results on the convergence of non-convex accelerated gradient descent in matrix factorization models. The technique is applied to matrix sensing problems with squared loss, for the estimation of a rank $r$ optimal solution $X^\star \in \mathbb{R}^{n imes n}$. We show that the acceleration leads to linear convergence rate, even under non-convex settings where the variable $X$ is represented as $U U^ op$ for $U \in \mathbb{R}^{n imes r}$. Our result has the same dependence on the condition number of the objective --and the optimal solution-- as that of the recent results on non-accelerated algorithms. However, acceleration is observed in practice, both in synthetic examples and in two real applications: neuronal multi-unit activities recovery from single electrode recordings, and quantum state tomography on quantum computing simulators.

Motivation & Objective

• To analyze the convergence behavior of accelerated gradient descent in non-convex matrix factorization problems.
• To establish theoretical guarantees for accelerated Procrustes Flow in low-rank matrix sensing with squared loss.
• To demonstrate that acceleration maintains the same condition-number dependence as non-accelerated methods while improving empirical performance.
• To validate the method on real-world applications such as neuronal multi-unit activity recovery and quantum state tomography.

Proposed method

• The method employs non-convex optimization via factorization $X = UU^\top$ for $U \in \mathbb{R}^{n \times r}$ to estimate a rank-$r$ solution $X^\star$.
• It applies accelerated gradient descent to the factorized variable $U$, leveraging momentum to improve convergence speed.
• Theoretical analysis focuses on matrix sensing with squared loss, proving linear convergence under non-convex settings.
• The convergence rate depends on the condition number of the objective and the optimal solution, matching non-accelerated results.
• The approach is validated through synthetic experiments and two real-world applications: single-electrode neuronal activity recovery and quantum state tomography.

Experimental results

Research questions

• RQ1 Does accelerated Procrustes Flow achieve linear convergence in non-convex low-rank matrix factorization?
• RQ2 How does the convergence rate of accelerated Procrustes Flow compare to non-accelerated methods in terms of condition number dependence?
• RQ3 Can accelerated Procrustes Flow be practically effective in real-world applications beyond synthetic settings?
• RQ4 What is the role of momentum in accelerating convergence for matrix sensing under non-convexity?

Key findings

• Accelerated Procrustes Flow achieves linear convergence in non-convex matrix factorization, even when $X = UU^\top$ is used to represent the solution.
• The convergence rate has the same dependence on the condition number as non-accelerated algorithms, indicating theoretical robustness.
• Practical acceleration is observed in synthetic experiments, confirming theoretical expectations.
• The method demonstrates improved performance in recovering neuronal multi-unit activities from single-electrode recordings.
• It also shows strong performance in quantum state tomography on quantum computing simulators, validating its real-world applicability.