[Paper Review] A Machine Learning Alternative to P-values

Min Lü, Hemant Ishwaran|arXiv (Cornell University)|Jan 18, 2017

Statistical Methods and Inference7 references3 citations

TL;DR

This paper proposes variable importance (VIMP) based on out-of-bag (OOB) prediction error as a robust alternative to p-values in regression models. By measuring how much a variable degrades model prediction when 'noised up', VIMP provides a scientifically interpretable, model-agnostic effect size that remains valid under model misspecification, outperforming p-values in reliability and interpretability for both parametric and nonparametric models.

ABSTRACT

This paper presents an alternative approach to p-values in regression settings. This approach, whose origins can be traced to machine learning, is based on the leave-one-out bootstrap for prediction error. In machine learning this is called the out-of-bag (OOB) error. To obtain the OOB error for a model, one draws a bootstrap sample and fits the model to the in-sample data. The out-of-sample prediction error for the model is obtained by calculating the prediction error for the model using the out-of-sample data. Repeating and averaging yields the OOB error, which represents a robust cross-validated estimate of the accuracy of the underlying model. By a simple modification to the bootstrap data involving "noising up" a variable, the OOB method yields a variable importance (VIMP) index, which directly measures how much a specific variable contributes to the prediction precision of a model. VIMP provides a scientifically interpretable measure of the effect size of a variable, we call the "predictive effect size", that holds whether the researcher's model is correct or not, unlike the p-value whose calculation is based on the assumed correctness of the model. We also discuss a marginal VIMP index, also easily calculated, which measures the marginal effect of a variable, or what we call "the discovery effect". The OOB procedure can be applied to both parametric and nonparametric regression models and requires only that the researcher can repeatedly fit their model to bootstrap and modified bootstrap data. We illustrate this approach on a survival data set involving patients with systolic heart failure and to a simulated survival data set where the model is incorrectly specified to illustrate its robustness to model misspecification.

Motivation & Objective

• To address the widespread misuse and limitations of p-values in regression analysis, especially their dependence on correct model assumptions.
• To provide a scientifically interpretable measure of effect size that is robust to model misspecification.
• To replace statistical significance with predictive importance using out-of-bag error from bootstrap resampling.
• To introduce two types of variable importance: VIMP for predictive effect size and marginal VIMP for discovery effect size.
• To demonstrate the method’s robustness and utility in both correctly and incorrectly specified models using real and simulated survival data.

Proposed method

• Uses the out-of-bag (OOB) error from bootstrap resampling as a cross-validated estimate of model prediction error.
• Applies a 'noising up' procedure to a variable by setting its coefficient to zero in bootstrap samples to assess its contribution to prediction error.
• Computes VIMP as the average increase in OOB prediction error when a variable is noised up, quantifying its predictive importance.
• Extends the method to marginal VIMP, which measures the incremental predictive gain of adding a variable to a model.
• Applies the OOB procedure to both parametric models (e.g., Cox regression) and nonparametric models (e.g., random survival forests).
• Uses B-splines to model non-linear effects and computes cluster-VIMP for groups of variables, such as spline terms for a single predictor.

Experimental results

Research questions

• RQ1 Can a variable importance measure based on prediction error serve as a reliable alternative to p-values in regression settings?
• RQ2 How does the proposed VIMP index perform under model misspecification, where p-values are invalid?
• RQ3 Can VIMP provide a scientifically interpretable effect size that is independent of model correctness?
• RQ4 How does marginal VIMP compare to traditional likelihood-ratio tests in identifying new predictive variables?
• RQ5 Can VIMP be effectively applied to both parametric and nonparametric models, including those with complex, non-linear relationships?

Key findings

• The VIMP index provides a robust, interpretable measure of predictive effect size that remains valid even when model assumptions are violated.
• In the simulated survival data with incorrect model specification, VIMP correctly identified tumor volume as highly predictive, while p-values failed to reflect this due to model misfit.
• After applying B-splines to model non-linear effects in tumor volume, the OOB model error decreased from 43% to 40%, and VIMP for tumor volume increased to 2.27, indicating improved predictive relevance.
• Marginal VIMP values of 4.20 for PSA and 2.27 for tumor volume were comparable across datasets, showing standardization across different data sets based on concordance index.
• The method successfully identified that tumor volume had a non-linear relationship with survival, which was missed by linear models but captured through B-spline-based VIMP.
• Cluster-VIMP allowed for the assessment of group effects, such as the combined predictive importance of multiple B-spline terms for a single variable, enhancing interpretability.