[Paper Review] A Theoretical Analysis of Sparse Recovery Stability of Dantzig Selector and LASSO

Yun-Bin Zhao, Duan Li|arXiv (Cornell University)|Nov 10, 2017

Sparse and Compressive Sensing Techniques43 references3 citations

TL;DR

This paper provides a deterministic theoretical analysis of sparse recovery stability for the Dantzig Selector and LASSO using a novel weak range space property assumption, which is less restrictive than traditional conditions like RIP or NSP. It establishes unified recovery error bounds with stability coefficients measured by Robinson’s constant, proving the assumption is both sufficient and necessary for standard Dantzig Selector stability.

ABSTRACT

Dantzig selector (DS) and LASSO problems have attracted plenty of attention in statistical learning, sparse data recovery and mathematical optimization. In this paper, we provide a theoretical analysis of the sparse recovery stability of these optimization problems in more general settings and from a new perspective. We establish recovery error bounds for these optimization problems under a mild assumption called weak range space property of a transposed design matrix. This assumption is less restrictive than the well known sparse recovery conditions such as restricted isometry property (RIP), null space property (NSP) or mutual coherence. In fact, our analysis indicates that this assumption is tight and cannot be relaxed for the standard DS problems in order to maintain their sparse recovery stability. As a result, a series of new stability results for DS and LASSO have been established under various matrix properties, including the RIP with constant $δ_{2k}< 1/\sqrt{2}$ and the (constant-free) standard NSP of order $k.$ We prove that these matrix properties can yield an identical recovery error bound for DS and LASSO with stability coefficients being measured by the so-called Robinson's constant, instead of the conventional RIP or NSP constant. To our knowledge, this is the first time that the stability results with such a unified feature are established for DS and LASSO problems. Different from the standard analysis in this area of research, our analysis is carried out deterministically, and the key analytic tools used in our analysis include the error bound of linear systems due to Hoffman and Robinson and polytope approximation of symmetric convex bodies due to Barvinok.

Motivation & Objective

• To develop a more general and less restrictive theoretical framework for analyzing sparse recovery stability of Dantzig Selector and LASSO.
• To replace classical conditions like restricted isometry property (RIP) and null space property (NSP) with a milder, more natural assumption: weak range space property of the transposed design matrix.
• To unify the stability analysis of Dantzig Selector and LASSO under a single theoretical framework with deterministic error bounds.
• To establish that the proposed weak range space property is not only sufficient but also necessary for standard Dantzig Selector stability.
• To express stability coefficients in terms of Robinson’s constant, offering a data-dependent, constant-free measure of stability.

Proposed method

• Uses a deterministic approach based on Hoffman’s error bound for linear systems to analyze solution stability.
• Applies Barvinok’s polytope approximation technique for symmetric convex bodies to handle geometric structure in the optimization problem.
• Introduces the weak range space property as a fundamental assumption, derived from the optimality conditions of convex optimization.
• Derives recovery error bounds via duality and norm-based estimation, using the conjugate norm and dual norm relationships.
• Establishes stability bounds through a chain of inequalities involving the ℓ1-norm of the error, the sparsity-promoting term σk(x)₁, and the parameter μ.
• Employs projection operators onto feasible sets and leverages Hausdorff distance between sets to bound solution deviation.

Experimental results

Research questions

• RQ1 Can sparse recovery stability for Dantzig Selector and LASSO be established under a less restrictive assumption than RIP, NSP, or mutual coherence?
• RQ2 Is the weak range space property of the transposed design matrix both necessary and sufficient for stable sparse recovery in standard Dantzig Selector problems?
• RQ3 Can a unified stability framework be developed for both Dantzig Selector and LASSO that uses a single type of stability coefficient?
• RQ4 How do the stability bounds for Dantzig Selector and LASSO compare when measured by Robinson’s constant instead of traditional RIP or NSP constants?
• RQ5 What is the role of the parameter μ in controlling the stability of the LASSO solution, particularly in relation to the ℓ1-norm constraint?

Key findings

• The weak range space property is a sufficient and necessary condition for the stability of the standard Dantzig Selector in sparse recovery.
• Recovery error bounds for both Dantzig Selector and LASSO are established under this assumption, with stability coefficients measured by Robinson’s constant.
• The paper proves that RIP with δ₂ₖ < 1/√2 and standard NSP of order k both imply the weak range space property, thus ensuring stability.
• The stability bounds are unified across Dantzig Selector and LASSO, with identical error expressions under the same matrix assumptions.
• For LASSO, when the solution lies on the boundary of the ℓ1-ball (‖x‖₁ = μ), the error bound is ‖x − x*‖₂ ≤ δ + 4γ̂[φ(Mᵀ(Ax−y)) + σₖ(x)₁/c], where δ is the Hausdorff distance and c is a constant.
• The analysis is fully deterministic and avoids probabilistic assumptions, offering stronger theoretical guarantees than standard compressed sensing approaches.