[Paper Review] Guarding from Spurious Discoveries in High Dimension

Jianqing Fan, Wen‐Xin Zhou|arXiv (Cornell University)|Dec 5, 2015

Statistical Methods and Inference25 references3 citations

TL;DR

This paper proposes a generalized maximum spurious correlation measure, computed via the LAMM algorithm, to quantify how well a response can be fitted by random covariate subsets under the null model. It derives the asymptotic distribution of this measure for generalized linear models and L1-regression, enabling consistent benchmarking via multiplier bootstrap to guard against spurious discoveries in high-dimensional data.

ABSTRACT

Many data-mining and statistical machine learning algorithms have been developed to select a subset of covariates to associate with a response variable. Spurious discoveries can easily arise in high-dimensional data analysis due to enormous possibilities of such selections. How can we know statistically our discoveries better than those by chance? In this paper, we define a measure of goodness of spurious fit, which shows how good a response variable can be fitted by an optimally selected subset of covariates under the null model, and propose a simple and effective LAMM algorithm to compute it. It coincides with the maximum spurious correlation for linear models and can be regarded as a generalized maximum spurious correlation. We derive the asymptotic distribution of such goodness of spurious fit for generalized linear models and $L_1$-regression. Such an asymptotic distribution depends on the sample size, ambient dimension, the number of variables used in the fit, and the covariance information. It can be consistently estimated by multiplier bootstrapping and used as a benchmark to guard against spurious discoveries. It can also be applied to model selection, which considers only candidate models with goodness of fits better than those by spurious fits. The theory and method are convincingly illustrated by simulated examples and an application to the binary outcomes from German Neuroblastoma Trials.

Motivation & Objective

• To address the pervasive problem of spurious discoveries in high-dimensional data analysis, where numerous covariate selections can falsely appear predictive.
• To define a statistically rigorous measure of 'goodness of spurious fit' that quantifies the best possible fit achievable by random covariate subsets under the null hypothesis.
• To develop a computationally efficient method (LAMM) to compute this spurious fit measure, generalizing the concept of maximum spurious correlation beyond linear models.
• To derive the asymptotic distribution of the goodness of spurious fit under generalized linear models and L1-regression, depending on sample size, dimensionality, and design covariance.
• To enable model selection by rejecting candidates whose fit is not significantly better than the benchmark spurious fit, thus guarding against false discoveries.

Proposed method

• Proposes a generalized maximum spurious correlation as a measure of goodness of spurious fit, extending beyond linear models to generalized linear models and L1-regularized regression.
• Introduces the LAMM algorithm (Lagrangian Augmented Method for Maximization) to compute the maximum spurious correlation efficiently under the null model.
• Derives the asymptotic distribution of the goodness of spurious fit, which depends on sample size, ambient dimension, number of covariates used, and design covariance structure.
• Employs multiplier bootstrap to consistently estimate the asymptotic distribution, enabling empirical calibration of statistical significance thresholds.
• Applies the benchmark to model selection by filtering out models whose fit is not substantially better than the spurious fit threshold.
• Validates the method through simulations and a real-world application to binary outcomes from German Neuroblastoma Trials.

Experimental results

Research questions

• RQ1 How can we statistically quantify the best possible fit achievable by a random subset of covariates when no true relationship exists between predictors and response?
• RQ2 What is the asymptotic distribution of the maximum spurious fit in generalized linear models and L1-regression under high-dimensional settings?
• RQ3 Can the asymptotic distribution of spurious fit be consistently estimated using multiplier bootstrap for practical inference?
• RQ4 How can the benchmark of spurious fit be used to improve model selection by filtering out false discoveries?
• RQ5 What is the empirical performance of the proposed method in detecting spurious associations in real-world high-dimensional data?

Key findings

• The proposed goodness of spurious fit measure generalizes the maximum spurious correlation beyond linear models, providing a unified framework for high-dimensional model evaluation.
• The asymptotic distribution of the goodness of spurious fit depends on sample size, ambient dimension, number of covariates in the fit, and the design covariance matrix.
• The asymptotic distribution can be consistently estimated using multiplier bootstrap, enabling practical implementation and inference.
• The benchmark derived from the spurious fit distribution effectively guards against false discoveries by setting a statistical threshold for model selection.
• Simulations and application to German Neuroblastoma Trial data demonstrate the method's ability to distinguish true signals from spurious correlations in high-dimensional settings.
• The LAMM algorithm efficiently computes the maximum spurious correlation, making the method scalable and applicable to real-world data analysis.