[Paper Review] A Selective Approach to Internal Inference

Samuel M. Gross, Jonathan Taylor+1 more|arXiv (Cornell University)|Oct 2, 2015

Statistical Methods and Inference11 references3 citations

TL;DR

This paper proposes a selective inference framework to fairly assess the predictive significance of biomarker signatures derived from high-dimensional gene expression data using the same dataset for both model building and inference. By accounting for selection bias through conditional inference on the active set of features, the method controls type I error and outperforms sample splitting in power, especially when the signature is learned via lasso with fixed lambda.

ABSTRACT

A common goal in modern biostatistics is to form a biomarker signature from high dimensional gene expression data that is predictive of some outcome of interest. After learning this biomarker signature, an important question to answer is how well it predicts the response compared to classical predictors. This is challenging, because the biomarker signature is an internal predictor -- one that has been learned using the same dataset on which we want to evaluate it's significance. We propose a new method for approaching this problem based on the technique of selective inference. Simulations show that our method is able to properly control the level of the test, and that in certain settings we have more power than sample splitting.

Motivation & Objective

• To address the challenge of performing valid statistical inference on internal predictors—biomarker signatures learned from the same data used for evaluation—where standard tests suffer from inflated type I error due to selection bias.
• To develop a method that fairly evaluates whether a high-dimensional biomarker signature provides predictive value beyond standard clinical predictors (e.g., age, weight), despite being derived from the same dataset.
• To improve upon existing approaches like sample splitting and pre-validation by using all data efficiently while maintaining proper error rate control.
• To extend selective inference techniques to the context of internal prediction, particularly for lasso-based signature selection with fixed lambda.
• To provide a practical, computationally feasible method for assessing the significance of internal predictors in high-dimensional settings common in modern biostatistics.

Proposed method

• The method uses selective inference to compute valid p-values for the significance of an internal predictor $\hat{y}$, which is derived from a lasso model on the same dataset used for inference.
• It conditions on the selection event—specifically, which features were selected by the lasso—by deriving the exact conditional distribution of the test statistic given the active set $\hat{E}$.
• The test statistic for $\hat{y}$ is based on a linear model $y = \theta\hat{y} + Z\gamma + \epsilon$, where $\theta$ is tested under the null $H_0: \theta = 0$.
• The method derives the joint distribution of the test statistic and the selection event, enabling exact p-values via sampling from the conditional distribution under the null.
• It also provides an analytical solution for testing the inclusion of the selected predictors $X_{\hat{E}}$, which avoids sampling and improves computational efficiency.
• Data carving is applied to reserve more data for testing when the hypothesis is promising, enhancing power without inflating type I error.

Experimental results

Research questions

• RQ1 Can selective inference be used to perform valid statistical inference on a biomarker signature that was learned from the same dataset used for evaluation?
• RQ2 How does the proposed method compare to sample splitting and pre-validation in terms of type I error control and statistical power?
• RQ3 Does the method maintain proper error rates when the predictor is derived via lasso with a fixed lambda, and how does it handle selection bias?
• RQ4 Can the method be extended to test the significance of the selected predictors $X_{\hat{E}}$ rather than just the fitted $\hat{y}$, and what are the advantages?
• RQ5 What is the impact of undersampling on the accuracy of p-values in the selective inference framework?

Key findings

• The proposed selective inference method properly controls the type I error rate at the nominal level (e.g., 0.05), as demonstrated in simulations under the null hypothesis.
• In simulations with no true effect, the method achieved a type I error rate of 0.05, while the naive t-test had a type I error of 0.999, confirming the necessity of correction.
• The method showed higher statistical power than sample splitting in settings where the biomarker signature was strongly predictive, due to more efficient use of data.
• When testing the inclusion of the selected predictors $X_{\hat{E}}$, the method provided a valid analytical solution, avoiding the need for costly sampling.
• Under-sampling in the Monte Carlo approximation of p-values led to high variance and invalid p-values, highlighting the need for sufficient sampling in practice.
• The method outperformed pre-validation and sample splitting in terms of error rate control and power, particularly in high-dimensional settings with strong signal.