[Paper Review] The Effect of Coherence on Sampling from Matrices with Orthonormal Columns, and Preconditioned Least Squares Problems

Ilse C. F. Ipsen, Thomas Wentworth|arXiv (Cornell University)|Mar 21, 2012

Sparse and Compressive Sensing Techniques21 references4 citations

TL;DR

This paper analyzes row sampling strategies—without replacement, with replacement, and Bernoulli sampling—for matrices with orthonormal columns, deriving condition number bounds in terms of coherence and leverage scores. It shows that sampling at least $\mathcal{O}(m\mu\ln n)$ rows ensures well-conditioned sampled matrices with high probability, and provides algorithms to generate matrices with user-defined coherence and leverage scores, validated via the kappa_SQ_v3 toolbox.

ABSTRACT

Motivated by the least squares solver Blendenpik, we investigate three strategies for uniform sampling of rows from m x n matrices Q with orthonormal columns. The goal is to determine, with high probability, how many rows are required so that the sampled matrices have full rank and are well-conditioned with respect to inversion. Extensive numerical experiments illustrate that the three sampling strategies (without replacement, with replacement, and Bernoulli sampling) behave almost identically, for small to moderate amounts of sampling. In particular, sampled matrices of full rank tend to have two-norm condition numbers of at most 10. We derive a bound on the condition number of the sampled matrices in terms of the coherence μ of Q. This bound applies to all three different sampling strategies; it implies a, not necessarily tight, lower bound of O(m μ ln(n)) for the number of sampled rows; and it is realistic and informative even for matrices of small dimension and the stringent requirement of a 99 percent success probability. For uniform sampling with replacement we derive a potentially tighter condition number bound in terms of the leverage scores of Q. To obtain a more easily computable version of this bound, in terms of just the largest leverage scores, we first derive a general bound on the two-norm of diagonally scaled matrices. To facilitate the numerical experiments and test the tightness of the bounds, we present algorithms to generate matrices with user-specified coherence and leverage scores. These algorithms, the three sampling strategies, and a large variety of condition number bounds are implemented in the Matlab toolbox kappa_SQ_v3.

Motivation & Objective

• To investigate the effect of coherence on the condition number of sampled submatrices from matrices with orthonormal columns.
• To evaluate three sampling strategies—without replacement, with replacement, and Bernoulli sampling—for their ability to produce well-conditioned, full-rank sampled matrices.
• To derive tight, computable bounds on the two-norm condition number of sampled matrices in terms of coherence and leverage scores.
• To develop algorithms for generating matrices with user-specified coherence and leverage scores for experimental validation.
• To implement and test the proposed bounds and sampling strategies in the kappa_SQ_v3 MATLAB toolbox.

Proposed method

• Derives a general bound on the two-norm of diagonally scaled matrices using majorization and eigenvalue-diagonal element relationships.
• Establishes a condition number bound for sampled matrices in terms of coherence $\mu$, valid for all three sampling strategies, implying a lower bound of $\mathcal{O}(m\mu\ln n)$ sampled rows for high success probability.
• For uniform sampling with replacement, derives a tighter condition number bound using leverage scores, and provides a simplified version based on the largest leverage scores.
• Introduces a novel algorithm to generate $m \times n$ matrices with orthonormal columns and user-specified coherence and leverage scores, based on majorization and symmetric matrix construction.
• Implements all sampling strategies, bounds, and matrix generation algorithms in the open-source MATLAB toolbox kappa_SQ_v3 for reproducible experiments.
• Validates theoretical bounds through extensive numerical experiments on matrices of varying dimensions and coherence levels.

Experimental results

Research questions

• RQ1 How does coherence $\mu$ influence the condition number of a randomly sampled submatrix from a matrix with orthonormal columns?
• RQ2 Do different sampling strategies—without replacement, with replacement, and Bernoulli—yield significantly different conditioning behavior for sampled matrices?
• RQ3 Can a tight, computable bound on the condition number of sampled matrices be derived in terms of coherence and leverage scores?
• RQ4 What is the minimum number of rows that must be sampled to ensure the sampled matrix is well-conditioned with high probability?
• RQ5 Can matrices with arbitrary user-specified coherence and leverage scores be algorithmically generated for experimental testing?

Key findings

• The condition number of the sampled matrix $SQ$ is bounded by $\mathcal{O}(m\mu\ln n)$ with high probability, providing a theoretical lower bound on the number of rows needed for well-conditioning.
• Numerical experiments show that all three sampling strategies—without replacement, with replacement, and Bernoulli—behave almost identically for small to moderate sampling levels.
• Sampled matrices of full rank tend to have two-norm condition numbers of at most 10, indicating strong numerical stability.
• A tighter condition number bound is derived for sampling with replacement, expressed in terms of leverage scores, which improves upon the coherence-based bound.
• The proposed algorithm generates matrices with user-specified coherence and leverage scores, enabling controlled experiments and validation of theoretical bounds.
• The kappa_SQ_v3 toolbox successfully implements all components, including matrix generation, sampling strategies, and condition number bounds, for reproducible research.