[Paper Review] Fast Low-Rank Matrix Learning with Nonconvex Regularization

Quanming Yao, James T. Kwok+1 more|arXiv (Cornell University)|Dec 3, 2015

Sparse and Compressive Sensing Techniques37 references17 citations

TL;DR

This paper proposes FaNCL, a fast algorithm for nonconvex low-rank matrix learning that accelerates proximal gradient methods by exploiting singular value thresholding and power methods to compute proximal operators efficiently on reduced subspaces. The method achieves significant speedups—over 10x faster than state-of-the-art solvers—while improving recovery accuracy and producing lower-rank solutions than nuclear norm regularization.

ABSTRACT

Low-rank modeling has a lot of important applications in machine learning, computer vision and social network analysis. While the matrix rank is often approximated by the convex nuclear norm, the use of nonconvex low-rank regularizers has demonstrated better recovery performance. However, the resultant optimization problem is much more challenging. A very recent state-of-the-art is based on the proximal gradient algorithm. However, it requires an expensive full SVD in each proximal step. In this paper, we show that for many commonly-used nonconvex low-rank regularizers, a cutoff can be derived to automatically threshold the singular values obtained from the proximal operator. This allows the use of power method to approximate the SVD efficiently. Besides, the proximal operator can be reduced to that of a much smaller matrix projected onto this leading subspace. Convergence, with a rate of O(1/T) where T is the number of iterations, can be guaranteed. Extensive experiments are performed on matrix completion and robust principal component analysis. The proposed method achieves significant speedup over the state-of-the-art. Moreover, the matrix solution obtained is more accurate and has a lower rank than that of the traditional nuclear norm regularizer.

Motivation & Objective

• To address the high computational cost of full SVD in proximal gradient methods for nonconvex low-rank matrix optimization.
• To improve recovery accuracy and lower-rank solutions compared to convex nuclear norm regularization.
• To develop an efficient algorithm that avoids expensive full SVD by leveraging structure in commonly used nonconvex regularizers.
• To exploit the 'sparse plus low-rank' structure in matrix completion to further accelerate computation.
• To ensure convergence to a critical point with a proven O(1/T) rate.

Proposed method

• Proposes a novel thresholding rule that automatically prunes small singular values in the proximal operator for common nonconvex regularizers like capped-ℓ1, LSP, TNN, SCAD, and MCP.
• Uses the power method to efficiently compute only the leading singular subspace, avoiding full SVD computation.
• Reduces the proximal operator computation to a smaller matrix projected onto the leading subspace, drastically reducing computational cost.
• Leverages the 'sparse plus low-rank' structure in matrix completion to further accelerate updates by exploiting sparsity in the difference between current iterate and observed entries.
• Applies proximal gradient descent with backtracking line search, ensuring convergence to a critical point at O(1/T) rate.
• Integrates these components into a unified algorithm, FaNCL, that maintains theoretical guarantees while achieving high efficiency.

Experimental results

Research questions

• RQ1 Can we avoid full SVD in proximal gradient methods for nonconvex low-rank matrix learning by identifying a thresholding rule for singular values?
• RQ2 Can the power method be effectively used to approximate the leading singular subspace efficiently, enabling faster computation of the proximal operator?
• RQ3 Does reducing the proximal operator computation to a smaller projected matrix significantly improve runtime without sacrificing accuracy?
• RQ4 Can the 'sparse plus low-rank' structure in matrix completion be exploited to further accelerate the algorithm?
• RQ5 Does the proposed method achieve better recovery accuracy and lower rank than nuclear norm regularization, while being significantly faster?

Key findings

• FamCL achieves over 10x speedup compared to the state-of-the-art GPG solver on video background removal tasks, with 68.3s vs. 1571.2s on the bootstrap video.
• On matrix completion, FaNCL reduces CPU time from 1009.3s (GPG) to 60.4s for capped-ℓ1 regularization, achieving the same 100% sparse support recovery.
• FaNCL achieves lower NMSE than nuclear norm regularization across all nonconvex regularizers, with NMSE values of 0.0029 (LSP) and 0.0033 (TNN) compared to 0.0041 for nuclear norm.
• In video denoising, FaNCL achieves higher PSNR (25.08 dB on bootstrap) than nuclear norm (23.01 dB), with a 10x speedup in computation time.
• The algorithm converges to a critical point with an O(1/T) convergence rate, matching theoretical guarantees of proximal gradient methods.
• The proposed method produces lower-rank solutions than nuclear norm regularization, demonstrating improved low-rank approximation quality.