[Paper Review] Alternating Direction Algorithms for $\ell_1$-Problems in Compressive Sensing

Junfeng Yang, Yin Zhang|ArXiv.org|Dec 7, 2009

Sparse and Compressive Sensing TechniquesEngineering47 references776 citations

TL;DR

This paper proposes first-order primal-dual algorithms based on the Alternating Direction Method (ADM) for solving $α$-norm minimization problems in compressive sensing, such as basis pursuit and denoising. By reformulating problems into partially separable forms and applying exact or inexact ADM, the algorithms achieve fast, stable, and robust convergence, especially under noisy conditions, outperforming state-of-the-art methods in relative error reduction and iteration efficiency.

ABSTRACT

In this paper, we propose and study the use of alternating direction algorithms for several $\ell_1$-norm minimization problems arising from sparse solution recovery in compressive sensing, including the basis pursuit problem, the basis-pursuit denoising problems of both unconstrained and constrained forms, as well as others. We present and investigate two classes of algorithms derived from either the primal or the dual forms of the $\ell_1$-problems. The construction of the algorithms consists of two main steps: (1) to reformulate an $\ell_1$-problem into one having partially separable objective functions by adding new variables and constraints; and (2) to apply an exact or inexact alternating direction method to the resulting problem. The derived alternating direction algorithms can be regarded as first-order primal-dual algorithms because both primal and dual variables are updated at each and every iteration. Convergence properties of these algorithms are established or restated when they already exist. Extensive numerical results in comparison with several state-of-the-art algorithms are given to demonstrate that the proposed algorithms are efficient, stable and robust. Moreover, we present numerical results to emphasize two practically important but perhaps overlooked points. One point is that algorithm speed should always be evaluated relative to appropriate solution accuracy; another is that whenever erroneous measurements possibly exist, the $\ell_1$-norm fidelity should be the fidelity of choice in compressive sensing.

Motivation & Objective

• To develop efficient, stable, and robust first-order algorithms for solving $ \ell_1$-minimization problems arising in compressive sensing.
• To address the challenge of sparse signal recovery from underdetermined linear systems with noisy or corrupted measurements.
• To demonstrate the superiority of the proposed ADM-based algorithms over existing state-of-the-art solvers in terms of convergence speed and solution accuracy.
• To highlight the importance of using $ \ell_1$-norm fidelity in the presence of erroneous measurements, advocating for its use over squared $ \ell_2$-norm.
• To provide a unified framework applicable to multiple $ \ell_1$-problems, including basis pursuit, denoising variants, and nonnegative counterparts.

Proposed method

• Reformulate $ \ell_1$-problems into partially separable forms by introducing auxiliary variables and constraints.
• Apply the alternating direction method (ADM) to the reformulated problems using augmented Lagrangian functions and alternating minimization.
• Derive two classes of algorithms: primal-based and dual-based, both updating primal and dual variables at each iteration.
• Use exact or inexact subproblem solves within the ADM framework, with per-iteration cost dominated by two matrix-vector multiplications.
• Implement the dual-based ADM as an exact method when the sensing matrix $A$ is orthonormal, improving efficiency.
• Develop a MATLAB package YALL1 to implement the proposed algorithms for eight different $ \ell_1$-models, including nonnegative variants.

Experimental results

Research questions

• RQ1 Can first-order primal-dual algorithms derived from the ADM framework achieve faster and more stable convergence for $ \ell_1$-problems in compressive sensing compared to state-of-the-art methods?
• RQ2 How does the performance of the proposed ADM algorithms vary under different noise levels and stopping tolerances in practical applications?
• RQ3 Why is the $ \ell_1$-norm fidelity term more effective than the $ \ell_2$-norm fidelity when measurements contain errors?
• RQ4 What is the impact of algorithmic parameters on convergence and solution accuracy, and how can robustness be maintained across different problem instances?
• RQ5 To what extent can the ADM framework be generalized to other $ \ell_1$-like regularized problems, such as matrix rank minimization or total variation regularization?

Key findings

• The proposed ADM algorithms converge faster and achieve lower relative errors than FPC-BB, SpaRSA, FISTA, CGD, SPGL1, and NESTA on noisy test problems.
• Dual-based ADM algorithms are generally more efficient than primal-based ones, especially when $A$ is orthonormal, due to exact subproblem solves.
• The algorithms are robust to changes in model and algorithmic parameters, showing consistent performance across diverse problem settings.
• The proposed algorithms achieve the best achievable solution accuracy under noisy conditions, outperforming other methods even when high-accuracy solutions are not required.
• The $ \ell_1$-norm fidelity in problem (6) acts as an exact penalty method, reducing to basis pursuit when $\nu$ is below a threshold, making it ideal for erroneous measurements.
• The YALL1 MATLAB package successfully implements the algorithms for eight $ \ell_1$-models, demonstrating broad applicability and practical utility.