[Paper Review] A Singular Value Thresholding Algorithm for Matrix Completion

Jian‐Feng Cai, Emmanuel J. Candès+1 more|ArXiv.org|Oct 18, 2008

Sparse and Compressive Sensing TechniquesEngineering34 references131 citations

TL;DR

This paper proposes the Singular Value Thresholding (SVT) algorithm for low-rank matrix completion via nuclear norm minimization, using iterative soft-thresholding of singular values to efficiently recover matrices from few observed entries. The method achieves fast convergence and low storage use, enabling recovery of 1,000×1,000 matrices in under a minute and billion-scale matrices from just 0.4% sampled entries.

ABSTRACT

This paper introduces a novel algorithm to approximate the matrix with minimum nuclear norm among all matrices obeying a set of convex constraints. This problem may be understood as the convex relaxation of a rank minimization problem, and arises in many important applications as in the task of recovering a large matrix from a small subset of its entries (the famous Netflix problem). Off-the-shelf algorithms such as interior point methods are not directly amenable to large problems of this kind with over a million unknown entries. This paper develops a simple first-order and easy-to-implement algorithm that is extremely efficient at addressing problems in which the optimal solution has low rank. The algorithm is iterative and produces a sequence of matrices (X^k, Y^k) and at each step, mainly performs a soft-thresholding operation on the singular values of the matrix Y^k. There are two remarkable features making this attractive for low-rank matrix completion problems. The first is that the soft-thresholding operation is applied to a sparse matrix; the second is that the rank of the iterates X^k is empirically nondecreasing. Both these facts allow the algorithm to make use of very minimal storage space and keep the computational cost of each iteration low. We provide numerical examples in which 1,000 by 1,000 matrices are recovered in less than a minute on a modest desktop computer. We also demonstrate that our approach is amenable to very large scale problems by recovering matrices of rank about 10 with nearly a billion unknowns from just about 0.4% of their sampled entries. Our methods are connected with linearized Bregman iterations for l1 minimization, and we develop a framework in which one can understand these algorithms in terms of well-known Lagrange multiplier algorithms.

Motivation & Objective

• To develop an efficient first-order algorithm for solving nuclear norm minimization problems arising in low-rank matrix completion.
• To address the computational intractability of large-scale matrix completion problems where interior-point methods fail due to high dimensionality.
• To exploit the low-rank structure of solutions to minimize storage and computational cost per iteration.
• To provide a practical and scalable alternative to semidefinite programming for matrix completion.
• To demonstrate the algorithm's effectiveness on both moderate and very large-scale problems, including those with nearly a billion unknowns.

Proposed method

• The algorithm uses an iterative scheme that alternates between a linear constraint projection and a singular value thresholding operation.
• At each iteration, the method applies soft-thresholding to the singular values of a matrix Y^k, defined as D_τ(Y^k), to produce the next iterate X^k.
• The singular value thresholding operator D_τ(Y) is computed using the Lanczos bidiagonalization with partial reorthogonalization, which is efficient for sparse inputs.
• The algorithm is derived from a dual formulation of nuclear norm minimization and is connected to Uzawa’s algorithm and linearized Bregman iterations.
• A continuation method is suggested, where τ is increased gradually using warm starts to improve convergence speed.
• The method maintains a nondecreasing rank in the iterates X^k, which supports low-rank structure and reduces computational overhead.

Experimental results

Research questions

• RQ1 Can a first-order, easy-to-implement algorithm achieve high efficiency and low storage usage for large-scale matrix completion problems?
• RQ2 Does the singular value thresholding operation remain computationally feasible when applied to sparse matrices in iterative schemes?
• RQ3 Can the rank of the iterates in the SVT algorithm be empirically shown to be nondecreasing and converge to the true rank of the solution?
• RQ4 How does the algorithm perform in the presence of noise, and can it achieve accurate recovery with limited sampling?
• RQ5 Can the SVT algorithm scale to problems with nearly a billion unknowns using only a fraction of the entries?

Key findings

• The SVT algorithm recovers a 1,000×1,000 matrix in less than a minute on a modest desktop computer, demonstrating high computational efficiency.
• The algorithm successfully recovers matrices of rank about 10 with nearly a billion unknowns from just 0.4% of their sampled entries, showing scalability to very large problems.
• In noisy settings, the algorithm converges in about 200 iterations to a tolerance of ε=1e-4, with a relative error of 0.0769, close to the noise ratio of 0.08.
• The rank of the iterates X^k is empirically nondecreasing and quickly reaches the true rank r of the unknown matrix M, supporting the algorithm’s low-rank exploitation.
• The computational cost per iteration remains low and stable, as the time for singular value thresholding does not increase significantly over iterations.
• Theoretical convergence is established, and the method is shown to be connected to well-known Lagrange multiplier algorithms, providing a solid theoretical foundation.