[Paper Review] Stochastic Variance-reduced Gradient Descent for Low-rank Matrix Recovery from Linear Measurements

Xiao Zhang, Lingxiao Wang+1 more|arXiv (Cornell University)|Jan 2, 2017

Sparse and Compressive Sensing Techniques40 references3 citations

TL;DR

This paper proposes a novel stochastic variance-reduced gradient descent algorithm for low-rank matrix recovery from linear measurements, leveraging matrix factorization and variance reduction to achieve linear convergence with lower computational complexity. It establishes optimal sample complexity and convergence rates—matching minimax statistical error in noisy settings and exact recovery in noiseless cases—outperforming state-of-the-art methods in both theory and practice.

ABSTRACT

We study the problem of estimating low-rank matrices from linear measurements (a.k.a., matrix sensing) through nonconvex optimization. We propose an efficient stochastic variance reduced gradient descent algorithm to solve a nonconvex optimization problem of matrix sensing. Our algorithm is applicable to both noisy and noiseless settings. In the case with noisy observations, we prove that our algorithm converges to the unknown low-rank matrix at a linear rate up to the minimax optimal statistical error. And in the noiseless setting, our algorithm is guaranteed to linearly converge to the unknown low-rank matrix and achieves exact recovery with optimal sample complexity. Most notably, the overall computational complexity of our proposed algorithm, which is defined as the iteration complexity times per iteration time complexity, is lower than the state-of-the-art algorithms based on gradient descent. Experiments on synthetic data corroborate the superiority of the proposed algorithm over the state-of-the-art algorithms.

Motivation & Objective

• To address the high computational cost of existing gradient-based methods for low-rank matrix sensing.
• To develop an efficient stochastic optimization algorithm that achieves linear convergence with reduced iteration and per-iteration complexity.
• To establish theoretical guarantees for both noisy and noiseless matrix recovery settings.
• To achieve optimal sample complexity and statistical error rates in low-rank matrix recovery.
• To demonstrate superior empirical performance on synthetic data compared to state-of-the-art algorithms.

Proposed method

• Formulates matrix sensing as a nonconvex optimization problem using matrix factorization: X = UV⊤, reducing computational cost.
• Reformulates the objective as a sum of n component functions ℓi(UV⊤) to enable stochastic optimization.
• Introduces a stochastic variance-reduced gradient descent algorithm inspired by Johnson and Zhang (2013), reducing gradient variance per iteration.
• Uses a mini-batch structure with b observations per component function to balance variance and computational load.
• Employs a reweighted gradient update strategy to improve convergence and stability.
• Applies theoretical tools such as restricted isometry property (RIP) and matrix perturbation bounds to analyze convergence.

Experimental results

Research questions

• RQ1 Can a stochastic variance-reduced gradient method achieve linear convergence for low-rank matrix recovery from linear measurements?
• RQ2 Does the proposed algorithm achieve the minimax optimal statistical error in the presence of noise?
• RQ3 Can the algorithm recover the true low-rank matrix exactly in the noiseless case with optimal sample complexity?
• RQ4 What is the computational complexity of the proposed method compared to existing gradient-based algorithms?
• RQ5 How does the algorithm perform empirically on synthetic data relative to state-of-the-art methods?

Key findings

• The algorithm converges linearly to the true low-rank matrix up to the minimax optimal statistical error in noisy settings, matching the lower bound O(rd′/N).
• In the noiseless case, the algorithm achieves exact recovery with optimal sample complexity O(rd′), matching theoretical limits.
• The overall computational complexity is O((Nd′2 + κ²bd′2)log(1/ϵ)), which is lower than state-of-the-art gradient-based methods.
• The algorithm achieves ϵ-accuracy with a number of iterations that scales logarithmically with 1/ϵ, indicating fast convergence.
• Empirical results on synthetic data show superior performance over existing algorithms in terms of convergence speed and accuracy.
• Theoretical analysis confirms that the method maintains stability and convergence under both noisy and noiseless conditions via rigorous bounds on gradient variance and Hessian approximation.