[Paper Review] Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems

Christopher De, Kunle Olukotun+1 more|arXiv (Cornell University)|Nov 4, 2014

Sparse and Compressive Sensing TechniquesEngineering40 references76 citations

TL;DR

This paper proposes Alecton, a stochastic gradient descent (SGD) variant with adaptive step sizes that ensures global convergence for non-convex low-rank matrix problems such as matrix completion, phase retrieval, and subspace tracking. It proves convergence within $O(\epsilon^{-1}n\log n)$ steps from random initialization under broad sampling conditions, leveraging a connection to stochastic power iteration and a novel martingale-based analysis.

ABSTRACT

Stochastic gradient descent (SGD) on a low-rank factorization is commonly employed to speed up matrix problems including matrix completion, subspace tracking, and SDP relaxation. In this paper, we exhibit a step size scheme for SGD on a low-rank least-squares problem, and we prove that, under broad sampling conditions, our method converges globally from a random starting point within $O(\\epsilon^{-1} n \\log n)$ steps with constant probability for constant-rank problems. Our modification of SGD relates it to stochastic power iteration. We also show experiments to illustrate the runtime and convergence of the algorithm.

Motivation & Objective

• To address the lack of global convergence guarantees for standard SGD in non-convex low-rank matrix problems such as matrix completion and subspace tracking.
• To develop an algorithm that converges globally from random initialization, avoiding expensive SVD-based initialization methods used in prior work.
• To provide convergence rates robust to various noise models, without requiring bounded noise magnitude assumptions.
• To establish a novel martingale-based analysis technique applicable to non-convex optimization, extending beyond prior work.

Proposed method

• Alecton modifies SGD by using a step size that scales inversely with the norm of the current iterate, ensuring stability and convergence.
• The algorithm is designed to emulate stochastic power iteration, linking it to eigenvalue problems and enabling global convergence.
• A key technical component is the use of the Restricted Isometry Property (RIP) for the measurement operator, with $3p$-RIP and parameter $\delta < 1/3$ ensuring strong convexity in a neighborhood of the solution.
• The method analyzes the Hessian of the transformed objective function, proving strong convexity in a ball of constant radius around the solution when $\|YY^T - A\|_F$ is small.
• A hybrid strategy is proposed: first use Alecton to reach a region of strong convexity, then switch to coordinate descent for linear-rate convergence to the exact solution.
• The analysis relies on a novel martingale-based technique that bounds the variance of stochastic gradients and enables convergence under general sampling conditions.

Experimental results

Research questions

• RQ1 Can stochastic gradient descent be globally convergent for non-convex low-rank matrix problems without requiring expensive initialization?
• RQ2 Does an adaptive step size rule exist that ensures convergence from random initialization in matrix completion and related problems?
• RQ3 Can the convergence analysis be made robust to general noise models without assuming bounded noise magnitude?
• RQ4 Is there a connection between Alecton and stochastic power iteration that enables global convergence in non-convex settings?
• RQ5 Can a martingale-based analysis technique be developed to establish global convergence rates in non-convex low-rank optimization?

Key findings

• Alecton achieves global convergence from random initialization in $O(\epsilon^{-1}n\log n)$ steps with constant probability for constant-rank problems.
• The algorithm ensures convergence even when standard SGD diverges, due to its adaptive step size rule based on the iterate norm.
• The method is robust to different noise models, as the analysis depends only on the variance of samples, not on bounded noise magnitude.
• The Hessian of the objective function is shown to be strongly convex in a ball of constant radius around the solution, enabling fast local convergence.
• The convergence rate is optimal in the general setting, as shown via a lower bound in Appendix E, despite being slower than some prior methods with expensive initialization.
• A hybrid approach combining Alecton and coordinate descent enables both fast global convergence and linear-rate refinement to the exact solution.