[Paper Review] Improving CUR Matrix Decomposition and the Nyström Approximation via Adaptive Sampling

Shusen Wang, Zhihua Zhang|arXiv (Cornell University)|Mar 18, 2013

Sparse and Compressive Sensing Techniques49 references149 citations

TL;DR

This paper proposes adaptive sampling algorithms for CUR matrix decomposition and the Nyström approximation that achieve improved relative-error bounds without restrictive assumptions on data matrices. By leveraging a general error bound for adaptive column/row sampling, the method ensures lower approximation error with low time complexity and reduced memory usage, outperforming standard and ensemble Nyström methods in theory and practice.

ABSTRACT

The CUR matrix decomposition and the Nyström approximation are two important low-rank matrix approximation techniques. The Nyström method approximates a symmetric positive semidefinite matrix in terms of a small number of its columns, while CUR approximates an arbitrary data matrix by a small number of its columns and rows. Thus, CUR decomposition can be regarded as an extension of the Nyström approximation. In this paper we establish a more general error bound for the adaptive column/row sampling algorithm, based on which we propose more accurate CUR and Nyström algorithms with expected relative-error bounds. The proposed CUR and Nyström algorithms also have low time complexity and can avoid maintaining the whole data matrix in RAM. In addition, we give theoretical analysis for the lower error bounds of the standard Nyström method and the ensemble Nyström method. The main theoretical results established in this paper are novel, and our analysis makes no special assumption on the data matrices.

Motivation & Objective

• Address the limitations of standard CUR and Nyström methods, which often suffer from high approximation error and lack of theoretical guarantees.
• Develop a more general error bound for adaptive column/row sampling that applies to arbitrary data matrices without special assumptions.
• Design new CUR and Nyström algorithms with expected relative-error bounds that improve accuracy over existing randomized methods.
• Ensure low time complexity and minimal RAM usage by avoiding full matrix storage, making the approach suitable for large-scale data.
• Provide theoretical lower bounds for standard and ensemble Nyström methods to establish the limits of performance for these techniques.

Proposed method

• Introduce a general error bound for adaptive sampling of columns and rows in matrix approximation, based on the leverage scores and spectral properties.
• Use this bound to design a new adaptive sampling strategy that selects columns and rows with higher probability based on their contribution to the low-rank structure.
• Construct the CUR decomposition by selecting $ c $ columns and $ r $ rows adaptively, then compute the middle matrix $ extbf{W} $ as the pseudo-inverse of the intersection of selected columns and rows.
• Apply the same adaptive sampling framework to the Nyström method, which approximates symmetric positive semidefinite matrices using a subset of columns.
• Introduce an ensemble Nyström method that averages $ t $ independent samples to improve stability and reduce variance.
• Derive theoretical bounds on the approximation error in Frobenius and nuclear norms, showing that the error scales with $ (1- heta) $, where $ heta $ controls sampling bias.

Experimental results

Research questions

• RQ1 Can adaptive sampling improve the relative-error bounds of CUR and Nyström approximations without assuming specific data structures?
• RQ2 What is the theoretical lower bound on the approximation error for standard and ensemble Nyström methods?
• RQ3 How does adaptive sampling compare to uniform or leverage-score-based sampling in terms of error and computational efficiency?
• RQ4 Can the proposed method achieve relative-error bounds while maintaining low time complexity and minimal memory footprint?
• RQ5 What is the impact of ensemble averaging on the stability and accuracy of Nyström approximations?

Key findings

• The proposed adaptive sampling algorithm achieves expected relative-error bounds for both CUR and Nyström approximations, with no special assumptions on the input matrix.
• The Frobenius norm error bound for the ensemble Nyström method is lower bounded by $ (1- heta)^2 igg{[}ig{(}m-2c+rac{c}{t}-kig{)}+kigg{(}rac{m-c+rac{c}{t}+krac{1- heta}{ heta}}{c+krac{1- heta}{ heta}}igg{)}^{2}igg{]} $, showing improved convergence with adaptive sampling.
• The nuclear norm error bound for the ensemble Nyström method is at least $ (1- heta)(m-c)rac{c+rac{1}{ heta}k}{c+rac{1- heta}{ heta}k} $, demonstrating strong theoretical guarantees.
• The paper establishes a lower bound on the relative-error ratio of the ensemble Nyström method, showing it can be as high as $ rac{m-c}{m-k}ig{(}1+rac{k}{c}ig{)} $ in the worst case.
• Theoretical analysis confirms that the standard Nyström method cannot achieve relative-error bounds in general, highlighting the advantage of adaptive sampling.
• The method avoids storing the full matrix in RAM and maintains low time complexity, making it suitable for large-scale and sparse matrices.