[Paper Review] Weighted Low Rank Matrix Approximation and Acceleration

Elena Tuzhilina, Trevor Hastie|arXiv (Cornell University)|Sep 22, 2021

Sparse and Compressive Sensing Techniques20 references4 citations

TL;DR

This paper proposes a weighted low-rank matrix approximation (WLRMA) framework that generalizes standard low-rank matrix approximation and matrix completion by incorporating element-wise weights. It introduces a proximal gradient algorithm with two acceleration techniques—Nesterov momentum and Anderson acceleration—demonstrating faster convergence on both synthetic and real-world MovieLens data, especially for low-rank solutions.

ABSTRACT

Low-rank matrix approximation is one of the central concepts in machine learning, with applications in dimension reduction, de-noising, multivariate statistical methodology, and many more. A recent extension to LRMA is called low-rank matrix completion (LRMC). It solves the LRMA problem when some observations are missing and is especially useful for recommender systems. In this paper, we consider an element-wise weighted generalization of LRMA. The resulting weighted low-rank matrix approximation technique therefore covers LRMC as a special case with binary weights. WLRMA has many applications. For example, it is an essential component of GLM optimization algorithms, where an exponential family is used to model the entries of a matrix, and the matrix of natural parameters admits a low-rank structure. We propose an algorithm for solving the weighted problem, as well as two acceleration techniques. Further, we develop a non-SVD modification of the proposed algorithm that is able to handle extremely high-dimensional data. We compare the performance of all the methods on a small simulation example as well as a real-data application.

Motivation & Objective

• To generalize low-rank matrix approximation (LRMA) and low-rank matrix completion (LRMC) by introducing element-wise weights, enabling broader applicability to problems with non-uniform confidence or precision in observations.
• To develop an efficient algorithm for solving the weighted LRMA problem, particularly in settings with missing data or varying observation reliability.
• To accelerate convergence of the baseline alternating least squares (ALS) algorithm using Nesterov momentum and Anderson acceleration, especially for high-dimensional or sparse matrices.
• To enable scalable computation via a non-SVD, skinny-matrix reformulation that avoids full SVDs and supports extremely high-dimensional data.

Proposed method

• Formulates WLRMA as a convex relaxation problem: minimize the weighted Frobenius norm of the residual plus a nuclear norm penalty, using a weight matrix to encode observation confidence or precision.
• Develops a proximal gradient descent algorithm to solve the WLRMA problem, updating the low-rank factor matrices $A$ and $B$ via alternating least squares with soft-thresholding.
• Introduces Nesterov acceleration by incorporating momentum from previous iterates to speed up convergence in the ALS framework.
• Applies Anderson acceleration by reformulating the WLRMA problem as a fixed-point iteration and using a Krylov subspace method to accelerate convergence.
• Implements a non-SVD variant that maintains only skinny factor matrices ($A imes k$, $B imes k$) and computes rank via sequential SVDs on these small matrices, reducing computational cost.
• Uses guarded Anderson acceleration to ensure sufficient decrease in the objective at each step, improving stability.

Experimental results

Research questions

• RQ1 Can a unified framework be developed to generalize both standard low-rank approximation and matrix completion by incorporating element-wise weights?
• RQ2 How can convergence speed be improved for iterative WLRMA solvers in high-dimensional or sparse settings?
• RQ3 Does Nesterov acceleration or Anderson acceleration outperform standard alternating least squares in terms of convergence speed and stability for WLRMA?
• RQ4 Can a non-SVD, skinny-matrix formulation be used to scale WLRMA to extremely high-dimensional data without full SVD computation?

Key findings

• The proposed WLRMA framework subsumes standard LRMA and LRMC as special cases, with binary weights for missing data and general weights for confidence-based modeling.
• Anderson acceleration with depth $m=3$ achieved faster convergence than baseline ALS and Nesterov acceleration for $ ho=5$ and $ ho=20$ rank solutions on the MovieLens dataset.
• For $ ho=70$ (higher rank), Nesterov acceleration slightly outperformed Anderson for $ ho=10^{-4}$ accuracy, but both accelerated methods converged significantly faster than baseline ALS.
• The non-SVD, skinny-matrix reformulation enabled efficient computation of matrix rank and low-rank approximation without full SVDs, reducing per-iteration cost.
• Guarded Anderson acceleration ensured monotonic decrease in the objective, improving stability and reliability in practice.
• On the MovieLens dataset ($6000 imes 4000$, 5% observed), all accelerated methods achieved $ ho=10^{-4}$ accuracy in under 5 minutes, with Anderson showing the best performance for low-rank solutions.