[Paper Review] Shannon Theoretic Limits on Noisy Compressive Sampling

Mehmet Akçakaya, Vahid Tarokh|ArXiv.org|Nov 2, 2007

Sparse and Compressive Sensing Techniques7 references18 citations

TL;DR

This paper establishes Shannon-theoretic limits for noisy compressive sampling of sparse signals, showing that O(L) measurements are necessary and sufficient for reliable recovery in the linear sparsity regime (L = Θ(M)) when using optimal, information-theoretic decoding. For sublinear sparsity (L = o(M)), O(L log(M−L)) measurements are required under various performance metrics, revealing a fundamental gap between theoretical limits and practical L1-regularized methods.

ABSTRACT

In this paper, we study the number of measurements required to recover a sparse signal in ${\mathbb C}^M$ with $L$ non-zero coefficients from compressed samples in the presence of noise. For a number of different recovery criteria, we prove that $O(L)$ (an asymptotically linear multiple of $L$) measurements are necessary and sufficient if $L$ grows linearly as a function of $M$. This improves on the existing literature that is mostly focused on variants of a specific recovery algorithm based on convex programming, for which $O(L\log(M-L))$ measurements are required. We also show that $O(L\log(M-L))$ measurements are required in the sublinear regime ($L = o(M)$).

Motivation & Objective

• To determine the fundamental limits on the number of measurements required to recover sparse signals from noisy compressive samples.
• To analyze the trade-off between sparsity level L, ambient dimension M, and measurement count N under different performance metrics.
• To compare theoretical limits with practical algorithms like L1 regularization (LASSO), which require O(L log(M−L)) measurements.
• To establish achievability and converse results for multiple error metrics, including 0-1 loss, partial support recovery, and energy recovery.
• To show that signal power P does not need to grow with N under certain performance metrics, improving on prior work.

Proposed method

• Uses a joint typicality-based decoder to model optimal recovery under noisy compressive sampling, enabling information-theoretic analysis.
• Analyzes three performance metrics: exact support recovery (0-1 loss), partial support recovery (fraction α of non-zero entries), and energy recovery (fraction 1−γ of total power).
• Applies large-deviation analysis and entropy bounds to derive achievable and converse rates for each metric.
• Derives asymptotic bounds using functions f(z), k(z), d(z), and related logarithmic expressions to characterize the behavior of error probabilities.
• Considers both linear sparsity (L = θ(M)) and sublinear sparsity (L = o(M)) regimes.
• Establishes converse results via contradiction and entropy-based bounds, showing that insufficient measurements lead to unreliable recovery.

Experimental results

Research questions

• RQ1 What is the minimum number of measurements N required to reliably recover a sparse signal with L non-zero coefficients from noisy compressive samples?
• RQ2 How do different performance metrics—exact support recovery, partial support recovery, and energy recovery—affect the required number of measurements?
• RQ3 Can O(L) measurements suffice for reliable recovery in the linear sparsity regime (L = θ(M)) under optimal decoding?
• RQ4 What is the fundamental gap between information-theoretic limits and practical L1-regularized methods (e.g., LASSO) in terms of measurement requirements?
• RQ5 Does the total signal power P need to grow with N for reliable recovery under different performance metrics?

Key findings

• For the linear sparsity regime (L = θ(M)), O(L) measurements are both necessary and sufficient for reliable recovery using an optimal, information-theoretic decoder.
• In the sublinear sparsity regime (L = o(M)), O(L log(M−L)) measurements are required for reliable recovery under all considered performance metrics.
• For Error Metric 1 (0-1 loss), reliable recovery is not possible if N ≺ C′₁ L log(M−L) for some constant C′₁ depending on μ(x) and ν.
• For Error Metric 2 (partial support recovery), reliable recovery is possible if N ≻ C′₃ L log(M−L) and P is constant, with C′₃ depending on α, μ(x), and ν.
• For Error Metric 3 (energy recovery), reliable recovery is possible if N ≻ C′₅ L log(M−L) when P is constant, with C′₅ depending on γ, P, and ν.
• The converse results show that O(L log(M−L)) is necessary for reliable recovery under all metrics when P is constant and L = o(M), indicating a fundamental limit for practical algorithms like LASSO.