[Paper Review] On Sparse Vector Recovery Performance in Structurally Orthogonal Matrices via LASSO

Chao-Kai Wen, Jun Zhang+3 more|arXiv (Cornell University)|Oct 27, 2014

Sparse and Compressive Sensing Techniques43 references3 citations

TL;DR

This paper analyzes the performance of LASSO in reconstructing sparse signals using structurally orthogonal measurement matrices—matrices formed by concatenating randomly orthogonal bases—via the replica method in the large-system limit. It derives an analytical mean-squared error (MSE) formula and demonstrates that such matrices achieve at least the same reconstruction performance as i.i.d. Gaussian matrices, making them highly suitable for large-scale compressive sensing applications due to fast computation and parallel processing support.

ABSTRACT

In this paper, we consider a compressed sensing problem of reconstructing a sparse signal from an undersampled set of noisy linear measurements. The regularized least squares or least absolute shrinkage and selection operator (LASSO) formulation is used for signal estimation. The measurement matrix is assumed to be constructed by concatenating several randomly orthogonal bases, referred to as structurally orthogonal matrices. Such measurement matrix is highly relevant to large-scale compressive sensing applications because it facilitates fast computation and also supports parallel processing. Using the replica method from statistical physics, we derive the mean-squared-error (MSE) formula of reconstruction over the structurally orthogonal matrix in the large-system regime. Extensive numerical experiments are provided to verify the analytical result. We then use the analytical result to study the MSE behaviors of LASSO over the structurally orthogonal matrix, with a particular focus on performance comparisons to matrices with independent and identically distributed (i.i.d.) Gaussian entries. We demonstrate that the structurally orthogonal matrices are at least as well performed as their i.i.d. Gaussian counterparts, and therefore the use of structurally orthogonal matrices is highly motivated in practical applications.

Motivation & Objective

• To analyze the mean-squared error (MSE) performance of LASSO in reconstructing sparse signals when using structurally orthogonal measurement matrices.
• To derive an analytical MSE expression for such matrices in the large-system regime using the replica method from statistical physics.
• To compare the reconstruction performance of structurally orthogonal matrices against i.i.d. Gaussian measurement matrices.
• To validate the analytical results through extensive numerical experiments.
• To establish the practical advantage of structurally orthogonal matrices in large-scale compressive sensing due to computational efficiency and parallelizability.

Proposed method

• The replica method is applied to derive the asymptotic MSE of LASSO reconstruction over structurally orthogonal matrices in the large-system limit (N → ∞).
• The measurement matrix is modeled as a concatenation of L randomly orthogonal bases, each represented by a Haar-distributed unitary matrix.
• The analysis incorporates complex-valued signals and uses the complex ℓ₁-norm in the LASSO formulation to improve performance over real-valued extensions.
• Gaussian integration and Hubbard-Stratonovich transformations are used to handle the partition function and average over random matrix ensembles.
• Saddle-point approximation is applied to simplify the resulting expressions, leading to a closed-form MSE expression involving the trace of matrix functions.
• The derivation extends prior results by accommodating non-identity covariance matrices (R_p) in the orthogonal bases, generalizing earlier work on i.i.d. or identity-structured matrices.

Experimental results

Research questions

• RQ1 How does the mean-squared error (MSE) of LASSO reconstruction behave over structurally orthogonal measurement matrices in the large-system limit?
• RQ2 What is the analytical MSE expression for LASSO when the measurement matrix is composed of concatenated orthogonal bases?
• RQ3 How does the performance of structurally orthogonal matrices compare to that of i.i.d. Gaussian matrices in terms of MSE for sparse signal recovery?
• RQ4 Under what conditions does the use of structurally orthogonal matrices yield superior or equivalent performance to i.i.d. Gaussian matrices?
• RQ5 Can the analytical MSE formula derived via the replica method be validated through numerical simulations?

Key findings

• The derived MSE formula for LASSO over structurally orthogonal matrices is analytically exact in the large-system limit and depends on the sparsity level, noise power, and the structure of the orthogonal bases.
• The analytical MSE expression is validated through extensive numerical experiments, showing strong agreement between theory and simulation across various sparsity and noise regimes.
• Structurally orthogonal matrices achieve at least the same reconstruction performance as i.i.d. Gaussian matrices, with no degradation in MSE under the same conditions.
• The performance equivalence or superiority of structurally orthogonal matrices is attributed to their favorable spectral properties and structured randomness, which maintain good incoherence with sparse signals.
• The results demonstrate that structurally orthogonal matrices are a strong candidate for large-scale compressive sensing due to their computational efficiency and support for parallel processing.
• The analysis confirms that the use of complex ℓ₁-norm in LASSO outperforms real-valued extensions, especially when real and imaginary parts of signals are correlated.