[Paper Review] A Unified Primal Dual Active Set Algorithm for Nonconvex Sparse Recovery

Jian Huang, Yuling Jiao+4 more|arXiv (Cornell University)|Oct 4, 2013

Sparse and Compressive Sensing Techniques3 citations

TL;DR

This paper proposes a unified primal-dual active set (UPDAS) algorithm for nonconvex sparse recovery problems using penalties such as ℓ⁰, bridge, SCAD, MCP, and capped-ℓ¹. By deriving a necessary optimality condition via thresholding operators and leveraging primal-dual variable coupling, the algorithm iteratively updates the active set and solves small-scale least-squares subproblems, achieving global convergence under restricted isometry conditions and demonstrating superior efficiency and accuracy in numerical experiments on simulated and real data.

ABSTRACT

In this paper, we consider the problem of recovering a sparse signal based on penalized least squares formulations. We develop a novel algorithm of primal-dual active set type for a class of nonconvex sparsity-promoting penalties, including $\ell^0$, bridge, smoothly clipped absolute deviation, capped $\ell^1$ and minimax concavity penalty. First we establish the existence of a global minimizer for the related optimization problems. Then we derive a novel necessary optimality condition for the global minimizer using the associated thresholding operator. The solutions to the optimality system are coordinate-wise minimizers, and under minor conditions, they are also local minimizers. Upon introducing the dual variable, the active set can be determined using the primal and dual variables together. Further, this relation lends itself to an iterative algorithm of active set type which at each step involves first updating the primal variable only on the active set and then updating the dual variable explicitly. When combined with a continuation strategy on the regularization parameter, the primal dual active set method is shown to converge globally to the underlying regression target under certain regularity conditions. Extensive numerical experiments with both simulated and real data demonstrate its superior performance in efficiency and accuracy compared with the existing sparse recovery methods.

Motivation & Objective

• To develop a unified, efficient algorithm for solving nonconvex sparse recovery problems with a broad class of sparsity-inducing penalties.
• To establish the existence of a global minimizer and derive a necessary optimality condition using thresholding operators for nonconvex penalties.
• To design an iterative algorithm that alternately updates primal and dual variables on the active set, ensuring local superlinear convergence.
• To prove global convergence of the algorithm under restricted isometry property (RIP) conditions on the design matrix.
• To demonstrate the method’s efficiency and accuracy through extensive numerical experiments on synthetic and real-world genetic data.

Proposed method

• The algorithm is based on a primal-dual active set framework, where the active set is determined jointly from primal and dual variables.
• At each iteration, the primal variable is updated only on the active set via a least-squares solution, significantly reducing computational cost.
• The dual variable is explicitly updated using a closed-form expression derived from the optimality system.
• A continuation strategy on the regularization parameter is employed to enhance convergence and solution accuracy.
• The method relies on a thresholding operator derived from the nonconvex penalty, which characterizes the necessary optimality condition.
• Theoretical analysis shows that solutions to the optimality system are coordinate-wise minimizers, and under mild conditions, they are also local minimizers.

Experimental results

Research questions

• RQ1 Can a unified algorithm be developed for multiple nonconvex sparsity-promoting penalties, including ℓ⁰, bridge, SCAD, MCP, and capped-ℓ¹?
• RQ2 What is the necessary optimality condition for global minimizers in nonconvex sparse recovery, and how can it be expressed via thresholding operators?
• RQ3 How can the active set be efficiently determined using both primal and dual variables in a nonconvex setting?
• RQ4 Under what conditions does the proposed primal-dual active set algorithm converge globally to the true solution?
• RQ5 How does the algorithm’s performance compare to existing methods in terms of accuracy and computational efficiency on real and simulated data?

Key findings

• The proposed UPDAS algorithm achieves global convergence to the underlying regression target under restricted isometry property (RIP) conditions on the design matrix.
• The algorithm exhibits local superlinear convergence due to solving small-scale least-squares problems on the active set at each iteration.
• Numerical experiments on simulated data show that UPDAS outperforms existing methods in both solution accuracy and computational speed.
• In a real-world genetic dataset (NFBC1966, n=5,123, p=9,114 SNPs), UPDAS identified 18 SNPs with ℓ⁰, 11 with ℓ¹/², 27 with SCAD/MCP, and 27 with capped-ℓ¹, showing strong consistency across penalties.
• The method successfully recovered known biological associations, such as rs3764261 and rs7499892 near the CETP gene linked to HDL cholesterol levels.
• The new tuning parameter selection rule, combined with continuation, further improves the accuracy and robustness of the algorithm in practice.