[Paper Review] Compressed Regression

Shuheng Zhou, John Lafferty+1 more|arXiv (Cornell University)|Jun 4, 2007

Sparse and Compressive Sensing Techniques19 references15 citations

TL;DR

This paper proposes compressed regression, a method that enables accurate sparse linear regression using randomly compressed data, preserving privacy through random linear projections. It establishes theoretical guarantees that ℓ1-regularized regression on compressed data achieves sparsistence (correct variable selection) and persistence (optimal prediction), while mutual information between compressed and original data decays to zero, ensuring information-theoretic privacy.

ABSTRACT

Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. In this paper we study a variant of this problem where the original n input variables are compressed by a random linear transformation to m<1-regularized compressed regression to identify the nonzero coefficients in the true model with probability approaching one, a property called “sparsistence.” In addition, we show that ℓ₁-regularized compressed regression asymptotically predicts as well as an oracle linear model, a property called “persistence.” Finally, we characterize the privacy properties of the compression procedure in information-theoretic terms, establishing upper bounds on the rate of information communicated between the compressed and uncompressed data that decay to zero.

Motivation & Objective

• To enable high-dimensional sparse regression on data that has been compressed via random linear transformations to reduce storage and computational costs.
• To ensure that the compressed data preserves the ability to identify the correct set of relevant predictors (sparsistence) and achieve optimal prediction performance (persistence).
• To provide information-theoretic privacy guarantees by showing that mutual information between original and compressed data decays to zero.
• To extend the theoretical foundations of compressed sensing and sparse regression to a privacy-preserving data transformation framework.
• To demonstrate that ℓ1-regularized regression on compressed data performs asymptotically as well as on original data, both in variable selection and prediction.

Proposed method

• Compress the original n × p data matrix X via a random m × n matrix 8, producing eX = 8X with m ≪ n.
• Apply ℓ1-regularized least squares (Lasso) to the compressed data: eβm = argmin_β (1/(2m))∥eY − eXβ∥²₂ + λm∥β∥₁.
• Use random affine transformations (eX = 8X + 1) to further enhance privacy, though the analysis focuses on linear compression.
• Establish theoretical conditions on m and λm to ensure sparsistence and persistence, relying on incoherence and concentration of measure under random projections.
• Analyze privacy using mutual information I(eX; X)/np, showing it decays to zero as m increases, even when the compression matrix 8 is known.
• Leverage results from compressed sensing and random matrix theory to bound the estimation error and recovery probability of the true sparse model.

Experimental results

Research questions

• RQ1 Can ℓ1-regularized regression on compressed data successfully recover the true set of nonzero coefficients (sparsistence) when m ≪ n?
• RQ2 Does compressed regression achieve prediction performance comparable to that on original data (persistence) under the same conditions?
• RQ3 What is the information-theoretic privacy guarantee of the compression procedure, particularly in terms of mutual information between original and compressed data?
• RQ4 How do the number of compressed samples m and the regularization parameter λm affect the recovery of the true sparse model?
• RQ5 Can the privacy guarantees be strengthened by using affine transformations instead of linear ones, and does this affect statistical performance?

Key findings

• Sparsistence is achieved when m satisfies C₁s² log(np) ≤ m ≤ s/(C₂n), and λm → 0 with mλ²m log p → ∞, ensuring the correct support is recovered with probability approaching one.
• Persistence holds when log₂(np) ≤ m ≤ n and the ℓ1-ball radius Ln,m = o((m/log(np))¹/⁴), guaranteeing that the predictive risk of the compressed estimator converges to the oracle risk.
• The mutual information between the original data X and the compressed data eX decays to zero as m increases, providing information-theoretic privacy even if the compression matrix 8 is known.
• Theoretical analysis shows that the inverse of the compressed Gram matrix (1/m)ZᵀS ZS is bounded in ∞-norm by 4/(3η) with high probability, enabling control over estimation error.
• The method remains robust to noise and maintains statistical consistency under random projections, with the recovery error decreasing as the number of compressed samples increases.
• Even with known compression matrices, the original data cannot be recovered from compressed data when m ≪ min(n, p), due to the highly underdetermined system.