[Paper Review] A Second-Order Method for Compressed Sensing Problems with Coherent and Redundant Dictionaries

John Dassios, Kimon Fountoulakis+1 more|arXiv (Cornell University)|May 16, 2014

Sparse and Compressive Sensing Techniques14 references6 citations

TL;DR

This paper proposes a primal-dual Newton conjugate gradients (pdNCG) method for solving compressed sensing problems with coherent and redundant dictionaries, leveraging second-order optimization with efficient preconditioning. It achieves global convergence and super-linear local convergence, outperforming first-order methods like TFOCS by up to 3.7× in speed on image reconstruction tasks while maintaining comparable reconstruction quality.

ABSTRACT

In this paper we are interested in the solution of Compressed Sensing (CS) problems where the signals to be recovered are sparse in coherent and redundant dictionaries. CS problems of this type are convex with non-smooth and non-separable regularization term, therefore a specialized solver is required. We propose a primal-dual Newton Conjugate Gradients (pdNCG) method. We prove global convergence and fast local rate of convergence for pdNCG. Moreover, well-known properties of CS problems are exploited for the development of provably effective preconditioning techniques that speed-up the approximate solution of linear systems which arise. Numerical results are presented on CS problems which demonstrate the performance of pdNCG compared to a state-of-the-art existing solver.

Motivation & Objective

• Address the challenge of solving compressed sensing problems involving coherent and redundant dictionaries, which lead to non-smooth, non-separable regularization terms.
• Develop a second-order optimization method that efficiently handles the ill-conditioning and complexity of such problems.
• Design effective, problem-specific preconditioners to accelerate the solution of linear systems arising in Newton iterations.
• Demonstrate superior performance over state-of-the-art first-order solvers like TFOCS in terms of convergence speed and robustness.
• Establish theoretical convergence guarantees for the proposed pdNCG method on $β$-regularized $β$-analysis problems.

Proposed method

• Propose a primal-dual Newton conjugate gradients (pdNCG) method tailored for $β$-analysis problems with coherent and redundant dictionaries.
• Use Newton's method in a primal-dual framework to solve the $β$-regularized least-squares problem: $\min_x \, c\|W^*x\|_1 + \frac{1}{2}\|Ax - b\|_2^2$.
• Approximate the solution of the Newton system using the conjugate gradient method, with a preconditioner derived from problem structure.
• Exploit known properties of compressed sensing, such as sparsity and the structure of $W$ (e.g., curvelets, total variation), to design effective, inexpensive preconditioners.
• Integrate the preconditioned CG solver into a globalization strategy to ensure global convergence of the pdNCG method.
• Leverage the W-RIP (Weighted Restricted Isometry Property) to justify the effectiveness of the preconditioning and convergence analysis.

Experimental results

Research questions

• RQ1 Can a second-order method achieve faster convergence than first-order methods for compressed sensing problems with coherent and redundant dictionaries?
• RQ2 How can effective preconditioning be designed for the linear systems arising in Newton iterations for such problems?
• RQ3 Does the proposed pdNCG method maintain global convergence while achieving super-linear local convergence?
• RQ4 To what extent does the method outperform state-of-the-art first-order solvers like TFOCS in terms of CPU time and reconstruction quality?
• RQ5 Can problem-specific structure (e.g., in iTV or curvelet frames) be exploited to design efficient, inexpensive preconditioners?

Key findings

• The pdNCG method achieves global convergence and a local super-linear rate of convergence for $β$-analysis compressed sensing problems with coherent and redundant dictionaries.
• On a synthetic Shepp-Logan phantom image reconstruction task, pdNCG was 3.7× faster than TFOCS, requiring 16.4 seconds vs. 60.7 seconds, with similar PSNR (17.8 dB).
• On single-pixel camera data, pdNCG was faster than TFOCS on four out of five images, requiring on average 1.4× less CPU time.
• The method demonstrated robust performance across diverse compressed sensing applications, including isotropic total variation (iTV) and curvelet-based reconstruction.
• Effective, problem-specific preconditioners were developed that significantly accelerated the conjugate gradient sub-solves, enhancing overall efficiency.
• Numerical results confirm that approximate solution of Newton systems is sufficient for acceleration, and that second-order methods can be competitive despite higher per-iteration cost.