[Paper Review] Perturbation Bootstrap in Adaptive Lasso

Debraj Das, Karl Gregory+1 more|arXiv (Cornell University)|Mar 9, 2017

Statistical Methods and Inference24 references3 citations

TL;DR

This paper proposes a modified perturbation bootstrap method for the Adaptive Lasso (Alasso) that achieves second-order correctness in high-dimensional settings, where the model dimension grows with sample size. Unlike the naive perturbation bootstrap of Minnier et al. (2011), which fails to correct for higher-order bias, the authors modify the bootstrap objective function and use studentization to ensure accurate distributional approximation of Alasso estimators, significantly improving inference accuracy over the oracle normal approximation.

ABSTRACT

The Adaptive Lasso(Alasso) was proposed by Zou [ extit{J. Amer. Statist. Assoc. extbf{101} (2006) 1418-1429}] as a modification of the Lasso for the purpose of simultaneous variable selection and estimation of the parameters in a linear regression model. Zou (2006) established that the Alasso estimator is variable-selection consistent as well as asymptotically Normal in the indices corresponding to the nonzero regression coefficients in certain fixed-dimensional settings. In an influential paper, Minnier, Tian and Cai [ extit{J. Amer. Statist. Assoc. extbf{106} (2011) 1371-1382}] proposed a perturbation bootstrap method and established its distributional consistency for the Alasso estimator in the fixed-dimensional setting. In this paper, however, we show that this (naive) perturbation bootstrap fails to achieve second order correctness in approximating the distribution of the Alasso estimator. We propose a modification to the perturbation bootstrap objective function and show that a suitably studentized version of our modified perturbation bootstrap Alasso estimator achieves second-order correctness even when the dimension of the model is allowed to grow to infinity with the sample size. As a consequence, inferences based on the modified perturbation bootstrap will be more accurate than the inferences based on the oracle Normal approximation. We give simulation studies demonstrating good finite-sample properties of our modified perturbation bootstrap method as well as an illustration of our method on a real data set.

Motivation & Objective

• To address the limitation of the naive perturbation bootstrap in achieving second-order correctness for the Adaptive Lasso estimator in high-dimensional settings.
• To develop a computationally efficient bootstrap method that accurately approximates the finite-sample distribution of Alasso estimators when the number of predictors grows with sample size.
• To provide a refined alternative to the oracle normal approximation for inference on non-zero regression coefficients in sparse high-dimensional linear models.
• To establish theoretical guarantees—specifically second-order correctness—under weak regularity conditions allowing for diverging dimensionality.

Proposed method

• Proposes a modified perturbation bootstrap objective function that adjusts for bias in the Alasso estimator by incorporating a corrected penalty term in the bootstrap loss function.
• Introduces a studentized version of the modified bootstrap estimator to ensure asymptotic distributional equivalence with the true sampling distribution of the Alasso estimator.
• Employs Edgeworth expansion techniques and higher-order asymptotic theory to analyze the convergence rate of the bootstrap distribution to the true sampling distribution.
• Uses Taylor expansions and martingale-type arguments to control estimation errors in the bootstrap variance and penalty terms under high-dimensional asymptotics.
• Applies Hoeffding’s and Bernstein’s inequalities to establish uniform concentration bounds for bootstrap statistics under weak moment conditions.
• Derives asymptotic equivalence between the Edgeworth expansions of the modified bootstrap and the true sampling distribution up to order $ o(n^{-1}) $, ensuring second-order correctness.

Experimental results

Research questions

• RQ1 Does the naive perturbation bootstrap method of Minnier et al. (2011) achieve second-order correctness in approximating the distribution of the Adaptive Lasso estimator in high-dimensional settings?
• RQ2 Can a modified perturbation bootstrap objective function be constructed to achieve second-order correctness when the model dimension grows with sample size?
• RQ3 How does the performance of the modified bootstrap compare to the oracle normal approximation in finite samples?
• RQ4 What are the theoretical conditions under which the modified bootstrap achieves second-order accuracy in high-dimensional linear models?

Key findings

• The naive perturbation bootstrap fails to achieve second-order correctness in approximating the distribution of the Adaptive Lasso estimator, even in fixed-dimensional settings, due to uncorrected bias from the penalty term.
• The proposed modified perturbation bootstrap objective function achieves second-order correctness in distributional approximation, even when the number of predictors grows with sample size.
• The modified bootstrap estimator, when properly studentized, matches the finite-sample distribution of the Alasso estimator up to $ o(n^{-1}) $ terms in Edgeworth expansion, ensuring higher accuracy in inference.
• Simulation studies demonstrate good finite-sample performance of the modified bootstrap, with coverage probabilities closer to nominal levels than those based on the oracle normal approximation.
• The method is computationally efficient and applicable in high-dimensional settings, making it a practical alternative to asymptotic normal approximations for inference on non-zero regression coefficients.
• Theoretical results confirm that the modified bootstrap distribution converges to the true sampling distribution at the same rate as the true estimator, with matching Edgeworth expansions up to $ o(n^{-1}) $, validating its use in inference.