[Paper Review] Functional Choice and Non-significance Regions in Regression

Laurie Davies|arXiv (Cornell University)|May 6, 2016

Advanced Statistical Methods and Models12 references3 citations

TL;DR

This paper introduces a non-significance testing framework for variable selection in regression, replacing traditional model selection criteria with a permutation-based approach that compares observed residual sums of squares to those from i.i.d. normal noise. The method uses simulation to compute p-values for subsets of covariates, identifying those not significantly better than random noise, and extends to non-parametric loss functions like L1 regression.

ABSTRACT

Given data $y$ and $k$ covariates $x$ the problem is to decide which covariates to include when approximating $y$ by a linear function of the covariates. The decision is based on replacing subsets of the covariates by i.i.d. normal random variables and comparing the error with that obtained by retaining the subsets. If the two errors are not significantly different for a particular subset it is concluded that the covariates in this subset are no better than random noise and they are not included in the linear approximation to $y$.

Motivation & Objective

• To address the challenge of selecting relevant covariates in regression when traditional model-based criteria (e.g., AIC, BIC) or regularization (e.g., Lasso) may be unreliable or require strong distributional assumptions.
• To develop a method that determines whether a subset of covariates contributes significantly to predicting the response variable beyond random noise.
• To construct non-significance regions for regression parameters using simulation-based inference, replacing confidence regions in settings where a 'true' parameter value is not assumed.
• To extend the approach beyond least squares to general discrepancy measures, including L1 regression, enhancing robustness to outliers and heavy-tailed errors.

Proposed method

• Replace each subset of excluded covariates with i.i.d. N(0,1) random variables and compute the resulting residual sum of squares (S²_e) under the same regression model.
• Compare the observed residual sum of squares (s²) from the full model to S²_e across many simulations to compute a p-value: p_e = proportion of simulations where S²_e < s².
• Declare a subset of covariates non-significant if p_e > 1 - α (e.g., α = 0.95), indicating they are no better than random noise.
• Use simulation to construct non-significance regions for regression parameters (e.g., median or L1 estimates), defining values not significantly different from the estimate.
• For the median, define a non-significance region as the set of values m for which the probability that perturbed data yields a smaller L1 loss than the observed median is at least 1 - α.
• Leverage asymptotic approximations involving chi-squared distributions and normal noise to derive the expected behavior of the discrepancy under perturbation, enabling efficient simulation.

Experimental results

Research questions

• RQ1 Can we determine whether a subset of covariates contributes significantly to predicting the response variable, beyond what would be expected from random noise?
• RQ2 How can we define a non-significance region for regression parameters (e.g., median) without assuming a 'true' value, using simulation-based inference?
• RQ3 To what extent does this method outperform or complement traditional model selection criteria like AIC or BIC in variable selection?
• RQ4 Can the method be generalized beyond least squares to other discrepancy measures such as L1 regression?
• RQ5 What is the asymptotic behavior of the discrepancy measure under random perturbations of the response variable, and how does it inform the construction of non-significance regions?

Key findings

• In the stack loss data, only the subset encoded as 3 (e = (1,1,0), excluding 'Acid.Conc') had a p-value of 0.231, indicating it was not significantly better than random noise.
• The 0.95 non-significance region for the median of the stack loss data was [11.86, 18.71], closely matching the 0.95 confidence region [11, 18] based on order statistics.
• The asymptotic approximation for the minimum L1 loss under perturbation is s₁(y) - χ²_k / (4f(0)), where f(0) is the density of the error distribution at zero.
• The method is robust and generalizable: it applies to any discrepancy measure d(y,x), including L1 and other M-estimators, not just least squares.
• The p-value for a subset e is defined as the proportion of simulations where the simulated residual sum of squares is less than the observed s², enabling formal significance testing.
• The approach avoids reliance on parametric assumptions about error distributions and provides a frequentist alternative to model selection criteria that do not require explicit model fitting.