[Paper Review] Iterative Hard Thresholding for Compressed Sensing

Thomas Blumensath, Mike E. Davies|ArXiv.org|May 5, 2008

Sparse and Compressive Sensing Techniques11 references93 citations

TL;DR

This paper introduces and analyzes the Iterative Hard Thresholding (IHT) algorithm for compressed sensing recovery, demonstrating it achieves near-optimal error guarantees, robustness to noise, and linear computational complexity per iteration. It proves IHTs matches CoSaMP in theoretical performance, requiring only matrix-vector applications of the sensing operator and its adjoint, with convergence in logarithmic iterations relative to signal-to-noise ratio.

ABSTRACT

Compressed sensing is a technique to sample compressible signals below the Nyquist rate, whilst still allowing near optimal reconstruction of the signal. In this paper we present a theoretical analysis of the iterative hard thresholding algorithm when applied to the compressed sensing recovery problem. We show that the algorithm has the following properties (made more precise in the main text of the paper) - It gives near-optimal error guarantees. - It is robust to observation noise. - It succeeds with a minimum number of observations. - It can be used with any sampling operator for which the operator and its adjoint can be computed. - The memory requirement is linear in the problem size. - Its computational complexity per iteration is of the same order as the application of the measurement operator or its adjoint. - It requires a fixed number of iterations depending only on the logarithm of a form of signal to noise ratio of the signal. - Its performance guarantees are uniform in that they only depend on properties of the sampling operator and signal sparsity.

Motivation & Objective

• To establish theoretical performance guarantees for the Iterative Hard Thresholding (IHTs) algorithm in compressed sensing.
• To show IHTs achieves error bounds comparable to state-of-the-art methods like CoSaMP and ℓ1-based approaches.
• To demonstrate that IHTs is robust to observation noise and requires only minimal, problem-size-linear memory.
• To prove that IHTs succeeds with a minimal number of observations, scaling linearly with sparsity and logarithmically with signal dimension.
• To show the algorithm's performance is uniform, depending only on the restricted isometry property and signal sparsity, not on coefficient magnitudes.

Proposed method

• The algorithm iteratively applies the adjoint of the sensing matrix to the residual, selects the largest s coefficients via hard thresholding, and updates the estimate.
• It uses a fixed-point iteration framework to minimize an ℓ0-regularized cost function, ensuring sparsity in each step.
• The method relies on the restricted isometry property (RIP) with constant δ3s < 0.5 to ensure stable recovery.
• Each iteration involves one application of the sensing matrix Φ and its transpose ΦT, ensuring low per-iteration complexity.
• The algorithm terminates after a bounded number of iterations, specifically O(log(‖ys‖₂ / ɛs)) iterations to achieve error < 6ɛs.
• A stopping criterion is derived based on the signal-to-noise ratio, ensuring estimation accuracy within a known error bound.

Experimental results

Research questions

• RQ1 Can the Iterative Hard Thresholding (IHTs) algorithm achieve theoretical recovery guarantees comparable to CoSaMP and ℓ1-based methods in compressed sensing?
• RQ2 How does the performance of IHTs scale with respect to noise, sparsity, and signal-to-noise ratio in terms of error bounds and iteration count?
• RQ3 What is the computational and memory complexity of IHTs, and how does it compare to other greedy algorithms in terms of efficiency?
• RQ4 To what extent are the performance guarantees of IHTs uniform, depending only on the restricted isometry constant and sparsity, not on coefficient distribution?
• RQ5 Why do numerical results sometimes show IHTs underperforming relative to other methods despite strong theoretical guarantees?

Key findings

• The IHTs algorithm achieves an estimation error of at most 6‖ẽ‖₂ within a finite number of iterations, matching the error bounds of CoSaMP and ℓ1 methods.
• The algorithm is robust to observation noise, with estimation error increasing linearly with noise magnitude.
• IHTs requires only O(s log N) observations, which is optimal up to a constant factor, matching the theoretical minimum for sparse recovery.
• The computational complexity per iteration is of the same order as applying the sensing operator or its adjoint, making it highly efficient.
• The number of iterations is bounded by O(log(‖ys‖₂ / ɛs)), depending logarithmically on the signal-to-noise ratio.
• The performance guarantees are uniform: they depend only on the restricted isometry constant δ3s and the sparsity level s, not on the magnitude or distribution of the largest coefficients.