[Paper Review] Regularization with the Smooth-Lasso procedure

Mohamed Hebiri|ArXiv.org|Mar 5, 2008

Statistical Methods and Inference36 references19 citations

TL;DR

This paper introduces the Smooth-Lasso (S-Lasso) estimator for high-dimensional linear regression, combining L1-penalization for sparsity with an l2-fused penalty to handle correlated predictors. It establishes theoretical consistency in variable selection and sparsity inequalities under high-dimensional asymptotics, outperforming Lasso and Elastic-Net in correlated design settings.

ABSTRACT

We consider the linear regression problem. We propose the S-Lasso procedure to estimate the unknown regression parameters. This estimator enjoys sparsity of the representation while taking into account correlation between successive covariates (or predictors). The study covers the case when $p\gg n$, i.e. the number of covariates is much larger than the number of observations. In the theoretical point of view, for fixed $p$, we establish asymptotic normality and consistency in variable selection results for our procedure. When $p\geq n$, we provide variable selection consistency results and show that the S-Lasso achieved a Sparsity Inequality, i.e., a bound in term of the number of non-zero components of the oracle vector. It appears that the S-Lasso has nice variable selection properties compared to its challengers. Furthermore, we provide an estimator of the effective degree of freedom of the S-Lasso estimator. A simulation study shows that the S-Lasso performs better than the Lasso as far as variable selection is concerned especially when high correlations between successive covariates exist. This procedure also appears to be a good challenger to the Elastic-Net (Zou and Hastie, 2005).

Motivation & Objective

• Address the limitation of the Lasso in selecting correlated predictors by incorporating a smoothness penalty.
• Develop a regularization procedure that maintains sparsity while improving selection performance in high-dimensional, correlated design settings.
• Establish theoretical consistency in variable selection and estimation error bounds for the S-Lasso under high-dimensional asymptotics (p ≫ n).
• Provide a theoretical framework for the effective degrees of freedom of the S-Lasso estimator.
• Demonstrate through simulations that S-Lasso outperforms Lasso and Elastic-Net in variable selection when predictors are correlated, especially in successive order.

Proposed method

• Propose the S-Lasso estimator as the solution to a convex optimization problem combining L1 and l2-fused penalties: ||Y - Xβ||_n² + λ||β||_1 + μ∑_{j=2}^p (β_j - β_{j-1})².
• Use the l2-fused penalty to encourage similarity between adjacent regression coefficients, improving selection in the presence of correlated covariates.
• Leverage the strict convexity of the l2-fused penalty to ensure unique solutions and facilitate optimization, unlike the l1-fused penalty in Fused-Lasso.
• Apply theoretical tools such as mutual coherence, oracle inequalities, and Stein’s lemma to derive finite-sample and asymptotic properties.
• Derive a sparsity inequality that bounds the estimation error in terms of the number of non-zero coefficients in the oracle model.
• Provide an estimator for the effective degrees of freedom of the S-Lasso, enabling model selection criteria like AIC/BIC in high-dimensional settings.

Experimental results

Research questions

• RQ1 Can a modification of the Lasso that incorporates smoothness between adjacent coefficients improve variable selection in high-dimensional, correlated design settings?
• RQ2 Does the S-Lasso achieve variable selection consistency under weaker conditions than the standard Lasso, particularly when predictors are correlated?
• RQ3 How does the S-Lasso perform in comparison to the Elastic-Net and Fused-Lasso in terms of estimation accuracy and sparsity?
• RQ4 What is the theoretical behavior of the S-Lasso in high-dimensional asymptotics (p ≫ n), particularly regarding estimation error and selection consistency?
• RQ5 Can an effective degrees of freedom estimator be derived for the S-Lasso to support model selection and inference?

Key findings

• The S-Lasso achieves variable selection consistency under conditions that are less restrictive than those required by the Lasso, especially when successive covariates are highly correlated.
• The S-Lasso satisfies a sparsity inequality, meaning the estimation error is bounded by a term proportional to the number of non-zero coefficients in the true model.
• Theoretical analysis shows that the S-Lasso estimator is consistent in estimation and selection under high-dimensional asymptotics (p ≫ n), provided tuning parameters λ_n and μ_n are chosen appropriately.
• The effective degrees of freedom of the S-Lasso are estimated via a continuous, piecewise constant function of the response vector, enabling model selection criteria.
• Simulation results demonstrate that the S-Lasso outperforms the Lasso in variable selection, particularly when successive covariates are correlated.
• The S-Lasso provides a favorable trade-off between sparsity (from L1 penalty) and smoothness (from l2-fused penalty), outperforming both Lasso and Elastic-Net in correlated design scenarios.