[Paper Review] Adaptive post-Dantzig estimation and prediction for non-sparse "large $p$ and small $n$" models

Lu Lin, Lixing Zhu+1 more|arXiv (Cornell University)|Aug 7, 2010

Statistical Methods and Inference19 references3 citations

TL;DR

This paper proposes adaptive post-Dantzig estimation for non-sparse high-dimensional models with large $p$ and small $n$, where traditional methods fail due to model sparsity assumptions. By combining the Dantzig selector with nonparametric adjustment and instrumental variables, the method achieves consistent, asymptotically normal estimation even under non-sparsity, significantly improving prediction accuracy over classical methods like the Gaussian Dantzig selector.

ABSTRACT

For consistency (even oracle properties) of estimation and model prediction, almost all existing methods of variable/feature selection critically depend on sparsity of models. However, for ``large $p$ and small $n$" models sparsity assumption is hard to check and particularly, when this assumption is violated, the consistency of all existing estimations is usually impossible because working models selected by existing methods such as the LASSO and the Dantzig selector are usually biased. To attack this problem, we in this paper propose adaptive post-Dantzig estimation and model prediction. Here the adaptability means that the consistency based on the newly proposed method is adaptive to non-sparsity of model, choice of shrinkage tuning parameter and dimension of predictor vector. The idea is that after a sub-model as a working model is determined by the Dantzig selector, we construct a globally unbiased sub-model by choosing suitable instrumental variables and nonparametric adjustment. The new estimation of the parameters in the sub-model can be of the asymptotic normality. The consistent estimator, together with the selected sub-model and adjusted model, improves model predictions. Simulation studies show that the new approach has the significant improvement of estimation and prediction accuracies over the Gaussian Dantzig selector and other classical methods have.

Motivation & Objective

• To address the inconsistency of existing estimation methods in non-sparse 'large $p$, small $n$' models where sparsity assumptions are violated.
• To develop a method that achieves estimation consistency and asymptotic normality regardless of model sparsity, tuning parameter choice, or dimensionality.
• To improve model prediction accuracy by correcting bias in working models selected by the Dantzig selector.
• To establish a framework for consistent inference in ultra-high-dimensional regression without relying on sparsity.

Proposed method

• The method begins with a working sub-model selected by the Dantzig selector to reduce dimensionality.
• It introduces instrumental variables to construct a globally unbiased sub-model, correcting bias from the initial selection.
• Nonparametric adjustment is applied to the selected sub-model using low-dimensional nonparametric estimation based on the instrumental variables.
• The final estimator is derived by solving a corrected estimating equation that incorporates the nonparametric adjustment and instrumental variable structure.
• Asymptotic normality and $\ell_2$ consistency are established under regularity conditions, even when the dimension $q$ of the parameter vector grows with sample size.
• The method adapts to non-sparsity, tuning parameter choice, and dimensionality, ensuring robust performance across diverse high-dimensional settings.

Experimental results

Research questions

• RQ1 Can consistent and asymptotically normal estimation be achieved in non-sparse 'large $p$, small $n$' models where sparsity assumptions fail?
• RQ2 How can bias from the Dantzig selector's working model be corrected to improve estimation and prediction accuracy?
• RQ3 What role do instrumental variables and nonparametric adjustment play in achieving consistency under non-sparsity?
• RQ4 Does the proposed method maintain consistency and asymptotic normality when the number of parameters $q$ increases with sample size?
• RQ5 How does the method compare in performance to the Gaussian Dantzig selector and other classical methods in non-sparse settings?

Key findings

• The proposed adaptive post-Dantzig estimator achieves $\|\hat{\theta} - \theta\|_{{\ell}_2}^2 = O_p(n^{-1})$ under fixed $q$, ensuring $\ell_2$ consistency.
• The estimator is asymptotically normal under regularity conditions, even when the dimension $q$ of the parameter vector diverges with sample size.
• The method achieves significant improvements in estimation and prediction accuracy over the Gaussian Dantzig selector and other classical methods in simulation studies.
• Nonparametric adjustment based on instrumental variables enables globally unbiased estimation, correcting the inherent bias of the Dantzig selector's working model.
• The method is adaptive to non-sparsity, tuning parameter choice, and dimensionality, making it robust across diverse high-dimensional settings.
• Theoretical results confirm that the estimation error is bounded by $O_p(h^k + 1/\sqrt{nh^{2(d+1)}}) + O_p(n^{-\mu})$, with optimal bandwidth choice yielding $O_p(n^{-k/(2(k+d+1))})$ convergence rate.