[Paper Review] Flexible Shrinkage Estimation in High-Dimensional Varying Coefficient Models

Heng Lian|arXiv (Cornell University)|Aug 13, 2010

Dam Engineering and Safety30 references3 citations

TL;DR

This paper proposes a double shrinkage estimation method using adaptive group Lasso on B-spline expansions to simultaneously perform variable selection and constant coefficient identification in high-dimensional varying coefficient models with $ p \gg n $. The method achieves oracle properties and consistency in model selection, even when the number of relevant variables grows with sample size, and establishes theoretical validity of a semiparametric BIC-type criterion for regularization parameter selection.

ABSTRACT

We consider the problem of simultaneous variable selection and constant coefficient identification in high-dimensional varying coefficient models based on B-spline basis expansion. Both objectives can be considered as some type of model selection problems and we show that they can be achieved by a double shrinkage strategy. We apply the adaptive group Lasso penalty in models involving a diverging number of covariates, which can be much larger than the sample size, but we assume the number of relevant variables is smaller than the sample size via model sparsity. Such so-called ultra-high dimensional settings are especially challenging in semiparametric models as we consider here and has not been dealt with before. Under suitable conditions, we show that consistency in terms of both variable selection and constant coefficient identification can be achieved, as well as the oracle property of the constant coefficients. Even in the case that the zero and constant coefficients are known a priori, our results appear to be new in that it reduces to semivarying coefficient models (a.k.a. partially linear varying coefficient models) with a diverging number of covariates. We also theoretically demonstrate the consistency of a semiparametric BIC-type criterion in this high-dimensional context, extending several previous results. The finite sample behavior of the estimator is evaluated by some Monte Carlo studies.

Motivation & Objective

• Address the challenge of simultaneous variable selection and constant coefficient identification in high-dimensional varying coefficient models where $ p \gg n $.
• Extend existing penalization methods to ultra-high-dimensional semiparametric models with a diverging number of covariates.
• Establish theoretical consistency for both variable selection and constant coefficient identification under sparsity assumptions.
• Develop and justify a semiparametric BIC-type criterion for automatic regularization parameter selection in high-dimensional settings.
• Demonstrate the method's consistency even when zero and constant coefficients are known a priori, reducing to semivarying coefficient models with diverging covariates.

Proposed method

• Use B-spline basis expansion to represent varying coefficients, enabling nonparametric estimation in a finite-dimensional approximation space.
• Apply adaptive group Lasso penalty to simultaneously shrink entire coefficient functions and identify constant coefficients via group structure.
• Formulate the estimation problem as a convex optimization problem to ensure global convergence and satisfy KKT conditions.
• Decompose the design matrix into components related to nonparametric and parametric subspaces to analyze asymptotic behavior.
• Leverage the Frobenius norm and matrix projection techniques to bound estimation errors and derive convergence rates.
• Introduce a semiparametric BIC-type criterion based on residual sum of squares and model complexity to select tuning parameters consistently.

Experimental results

Research questions

• RQ1 Can adaptive group Lasso be used to achieve simultaneous variable selection and constant coefficient identification in high-dimensional varying coefficient models?
• RQ2 Does the proposed method maintain the oracle property for constant coefficients under ultra-high-dimensional settings with $ p \gg n $?
• RQ3 Is the semiparametric BIC-type criterion consistent in selecting regularization parameters when the number of covariates diverges?
• RQ4 How does the method perform in finite samples, particularly in distinguishing between zero, constant, and nonparametric coefficients?
• RQ5 Can the theoretical consistency results be extended to semivarying coefficient models with a diverging number of covariates?

Key findings

• The proposed double shrinkage estimator achieves model selection consistency, correctly identifying zero coefficients with probability tending to one.
• The method achieves the oracle property for constant coefficients, meaning they are estimated as efficiently as if their true structure were known.
• Consistency of the semiparametric BIC-type criterion is established, enabling automatic and consistent selection of regularization parameters in high-dimensional settings.
• Theoretical bounds show that estimation error converges to zero at a rate of $ O_p(\sqrt{ns}/n) $, with additional terms controlled via B-spline approximation and design matrix decomposition.
• Finite sample simulations confirm the method’s ability to accurately distinguish between zero, constant, and varying coefficients in high-dimensional scenarios.
• Even when the true zero and constant coefficient structure is known, the method’s consistency results are novel and extend to semivarying coefficient models with diverging covariates.