[Paper Review] Stepwise Choice of Covariates in High Dimensional Regression

Laurie Davies|arXiv (Cornell University)|Oct 17, 2016

Statistical Methods and Inference11 references3 citations

TL;DR

This paper proposes a model-free stepwise regression method for high-dimensional linear models that selects covariates by comparing their reduction in sum of squared residuals to that of independent Gaussian noise. It derives exact p-values using beta distributions to assess significance without regularization, cross-validation, or error variance estimation, achieving consistency and improved performance over lasso in simulations.

ABSTRACT

Given data y(n) and p(n)covariates x(n) one problem in linear regression is to decide which if any of the covariates to include. There are many articles on this problem but all are based on a stochastic model for the data. This paper gives what seems to be a new approach which does not require any form of model. It is conceptually and algorithmically simple and consistency results can be proved under appropriate assumptions.

Motivation & Objective

• To develop a covariate selection method in high-dimensional linear regression that does not rely on stochastic models or error variance estimation.
• To provide a computationally simple and invariant alternative to existing methods like lasso, which require regularization or cross-validation.
• To establish a significance testing framework based on comparing actual covariates to i.i.d. Gaussian noise in terms of residual reduction.
• To extend the method to robust and non-linear regression with approximated p-values.
• To demonstrate consistency and improved false discovery control in high-dimensional settings.

Proposed method

• The method computes the probability that a random Gaussian covariate would reduce the sum of squared residuals more than the actual covariate, using the exact distribution of residual reduction under noise.
• It uses the exact distribution $ SS_{ u} \stackrel{D}{=} ss_0(1 - B_{1/2, (n - \nu_0 - 1)/2}) $, where $ B_{a,b} $ is a beta random variable, to compute p-values.
• The p-value for including a new covariate is given by $ \mathbf{P}(SS_{01} \leq ss_{01}) = 1 - \text{pbeta}(1 - ss_{01}/ss_0, 1/2, (n - \nu_0 - 1)/2)^{q(n) - \nu_0} $.
• The stopping rule is derived from inverting this p-value: $ ss_{01} > ss_0 \left(1 - \text{qbeta}((1 - \alpha)^{1/(q(n) - \nu_0)}, 1/2, (n - \nu_0 - 1)/2) \right) $.
• The method is invariant under affine transformations and permutations of covariates, and avoids multiple testing issues via joint probability $ \prod_{j=1}^{\nu} (1 - p_j) $.
• For non-linear and robust regression, approximations based on chi-squared distributions are used when exact p-values are unavailable.

Experimental results

Research questions

• RQ1 Can covariate selection in high-dimensional regression be performed without assuming a linear model or estimating error variance?
• RQ2 What is the exact probability that a random Gaussian covariate would outperform an actual covariate in reducing residual sum of squares?
• RQ3 How can a stepwise selection procedure be constructed that is consistent and free from regularization or cross-validation?
• RQ4 Does the method control the false discovery rate better than lasso in high-dimensional settings?
• RQ5 Can the approach be extended to robust and non-linear regression with minimal assumptions?

Key findings

• The method achieves consistency in high-dimensional linear regression under appropriate regularity conditions, with no need for error variance estimation.
• The exact p-value for covariate inclusion is derived using the beta distribution, replacing earlier chi-squared approximations.
• In the leukemia dataset with n=72 and q(n)=3571, the method correctly identified relevant genes with fewer false positives than lasso.
• In graphical model reconstruction, 1109 out of 1747 true edges were recovered with only two false positives, outperforming lasso-based methods.
• Simulations show the method maintains low false discovery rates and better edge recovery than lasso, even under high-dimensional settings.
• The asymptotic stopping rule is $ ss_{01} > ss_0 \left(1 - \frac{2\log q(n) - \log\log q(n) - 2\log(-\log(1 - \alpha))}{n} \right) $, providing a model-free threshold.