[Paper Review] Sparsistency of $\ell_1$-Regularized $M$-Estimators

Yen-Huan Li, Jonathan Scarlett+2 more|arXiv (Cornell University)|Oct 28, 2014

Sparse and Compressive Sensing Techniques20 references3 citations

TL;DR

This paper introduces the Local Structured Smoothness Condition (LSSC) to establish sparsistency—consistent recovery of the true sparsity pattern—for a broad class of $μathtt{1}$-regularized $M$-estimators in high-dimensional models. By verifying LSSC for models like logistic regression, gamma regression, and graphical models, the authors derive unified, deterministic sufficient conditions on the regularization parameter, sample size, dimension, and sparsity level, ensuring high-probability support recovery under general conditions.

ABSTRACT

We consider the model selection consistency or sparsistency of a broad set of $\ell_1$-regularized $M$-estimators for linear and non-linear statistical models in a unified fashion. For this purpose, we propose the local structured smoothness condition (LSSC) on the loss function. We provide a general result giving deterministic sufficient conditions for sparsistency in terms of the regularization parameter, ambient dimension, sparsity level, and number of measurements. We show that several important statistical models have $M$-estimators that indeed satisfy the LSSC, and as a result, the sparsistency guarantees for the corresponding $\ell_1$-regularized $M$-estimators can be derived as simple applications of our main theorem.

Motivation & Objective

• To establish model selection consistency (sparsistency) for a broad class of $μathtt{1}$-regularized $M$-estimators in high-dimensional statistical models.
• To identify general, checkable conditions on loss functions that ensure reliable recovery of the true sparse parameter support.
• To unify existing sparsistency results for specific models (e.g., logistic regression, Gaussian graphical models) under a single theoretical framework.
• To derive deterministic, non-asymptotic sufficient conditions for sparsistency in terms of the regularization parameter, ambient dimension, sparsity level, and sample size.
• To enable sample complexity bounds in high-dimensional regimes where $p$ and $s$ grow with $n$, including diverging dimensions.

Proposed method

• Introduce the Local Structured Smoothness Condition (LSSC) as a key technical assumption on the loss function, controlling its smoothness in a structured neighborhood around the true parameter.
• Formulate the $μathtt{1}$-regularized $M$-estimator as minimizing a convex loss function plus an $μathtt{1}$-norm penalty, with the goal of recovering the true support of the sparse parameter.
• Derive deterministic sufficient conditions for sparsistency based on the LSSC, the regularization parameter $τ_n$, and the triple $(p, n, s)$, using concentration inequalities and convex analysis.
• Apply the main theorem to specific models—logistic regression, gamma regression, Gaussian graphical models, and generalized linear models—by verifying that their loss functions satisfy the LSSC.
• Use Bernstein’s inequality and union bounds to control the sub-Gaussian-like behavior of the gradient of the empirical loss at the true parameter, ensuring it is bounded with high probability.
• Leverage self-concordance and positive definiteness of the Hessian on the true support to guarantee uniqueness and stability of the estimator in the constrained subspace.

Experimental results

Research questions

• RQ1 Under what general conditions on the loss function can $μathtt{1}$-regularized $M$-estimators consistently recover the true sparsity pattern in high-dimensional models?
• RQ2 Can a single theoretical framework unify sparsistency guarantees across diverse models such as logistic regression, gamma regression, and graphical models?
• RQ3 What are the minimal, checkable assumptions on the loss function that ensure sparsistency, particularly in high-dimensional settings with diverging $p$ and $s$?
• RQ4 How do the required sample size and regularization parameter scale with the ambient dimension $p$, sparsity level $s$, and number of samples $n$?
• RQ5 Can the theoretical guarantees be extended to non-linear models beyond linear or Gaussian settings, such as exponential family models?

Key findings

• The Local Structured Smoothness Condition (LSSC) is a sufficient condition that ensures sparsistency of $μathtt{1}$-regularized $M$-estimators under general convex loss functions.
• The main theorem provides deterministic, non-asymptotic sufficient conditions for sparsistency in terms of the regularization parameter $τ_n$, ambient dimension $p$, sparsity level $s$, and sample size $n$, with explicit scaling laws.
• For logistic regression, gamma regression, and Gaussian graphical models, the LSSC is verified, enabling direct application of the main theorem to derive sample complexity bounds.
• The paper establishes that the required regularization parameter $τ_n$ scales as $O(\sqrt{\frac{\log p}{n}})$ under standard assumptions, ensuring high-probability support recovery.
• The analysis allows for diverging dimensions, with $p$ growing exponentially with $n$, and supports scaling laws where $s^2(\log p)\nu_n^2 \ll n$ suffices for consistency.
• The results generalize prior work by avoiding hard-to-verify assumptions, offering a more practical and widely applicable framework for sparsistency analysis in high-dimensional statistics.