[Paper Review] False Discovery Rate Control via Data Splitting

Chenguang Dai, Buyu Lin+2 more|arXiv (Cornell University)|Feb 20, 2020

Statistical Methods and Inference64 references8 citations

TL;DR

This paper proposes a data-splitting framework for false discovery rate (FDR) control in high-dimensional feature selection, using independent estimates of regression coefficients to ensure symmetry under the null. Multiple Data Splitting (MDS) stabilizes selection, boosts power, and reduces false discovery proportion variance, achieving asymptotic FDR control in linear and Gaussian graphical models with minimal distributional assumptions.

ABSTRACT

Selecting relevant features associated with a given response variable is an important problem in many scientific fields. Quantifying quality and uncertainty of a selection result via false discovery rate (FDR) control has been of recent interest. This article introduces a data-splitting method (referred to as “DS”) to asymptotically control the FDR while maintaining a high power. For each feature, DS constructs a test statistic by estimating two independent regression coefficients via data splitting. FDR control is achieved by taking advantage of the statistic’s property that, for any null feature, its sampling distribution is symmetric about zero; whereas for a relevant feature, its sampling distribution has a positive mean. Furthermore, a Multiple Data Splitting (MDS) method is proposed to stabilize the selection result and boost the power. Surprisingly, with the FDR under control, MDS not only helps overcome the power loss caused by data splitting, but also results in a lower variance of the false discovery proportion (FDP) compared with all other methods in consideration. Extensive simulation studies and a real-data application show that the proposed methods are robust to the unknown distribution of features, easy to implement and computationally efficient, and are often the most powerful ones among competitors especially when the signals are weak and correlations or partial correlations among features are high. Supplementary materials for this article are available online.

Motivation & Objective

• To address the challenge of controlling the false discovery rate (FDR) in high-dimensional feature selection with minimal assumptions on feature distributions.
• To overcome the power loss typically associated with data splitting in high-dimensional inference.
• To develop a robust, computationally efficient method for FDR control that maintains high statistical power even under high correlation and weak signals.
• To extend FDR control to settings where the joint distribution of features is unknown or complex.
• To provide a framework that is easy to implement using standard high-dimensional regression tools and requires no knowledge of the true conditional distribution of features.

Proposed method

• Split the data into two independent parts: one for feature selection (e.g., via Lasso) and one for estimating the test statistic (e.g., OLS on the remaining data).
• For each feature, compute a test statistic as the difference between two independent regression coefficient estimates obtained from separate data splits.
• Leverage the symmetry of the test statistic's sampling distribution under the null (i.e., symmetric about zero for null features) to enable FDR control.
• Implement Multiple Data Splitting (MDS) by repeating the data-splitting and selection process multiple times and aggregating results to stabilize selection and improve power.
• Use the Benjamini-Hochberg procedure (BHq) on the aggregated test statistics to control the FDR at a pre-specified level.
• Ensure theoretical FDR control by relying on the asymptotic normality and independence of the coefficient estimates under the null.

Experimental results

Research questions

• RQ1 Can data-splitting strategies be used to asymptotically control the FDR in high-dimensional linear and Gaussian graphical models?
• RQ2 Does Multiple Data Splitting (MDS) reduce the variance of the false discovery proportion (FDP) compared to single-splitting methods?
• RQ3 Can the proposed method maintain high power when signals are weak and features are highly correlated?
• RQ4 How does the method perform in practice when the joint distribution of features is unknown or misspecified?
• RQ5 Can the framework be extended to nonlinear models or complex data types such as images and natural language?

Key findings

• The proposed data-splitting method asymptotically controls the FDR at any designated level for linear and Gaussian graphical models in both low and high dimensions.
• MDS significantly reduces the variance of the false discovery proportion (FDP) compared to all other methods considered, including single data splitting and knockoff filtering.
• MDS improves statistical power, especially under high correlation and weak signals, outperforming BHq, knockoff filtering, and DeepPINK in multiple simulation and real-data settings.
• In a real-world HIV drug resistance study, MDS discovered more true mutations than competitors for five out of seven PI drugs and four out of six NRTI drugs, with superior performance on drugs like DDI and TDF.
• The method is robust to unknown feature distributions and does not require knowledge of the joint distribution of features, unlike model-X knockoff filtering.
• The framework is computationally efficient and easy to implement using standard high-dimensional regression software, making it accessible for practical use.