[Paper Review] BONuS: Multiple multivariate testing with a data-adaptivetest statistic

Chiao-Yu Yang, Lihua Lei+2 more|arXiv (Cornell University)|Jun 29, 2021

Statistical Methods in Clinical Trials33 references4 citations

TL;DR

BONuS is a data-adaptive empirical Bayes framework for multiple multivariate testing that improves statistical power by learning an optimal p-value transformation from the full dataset, while rigorously controlling the false discovery rate (FDR) in finite samples—even under model misspecification. It uses a masked data scheme to enable interactive model building without compromising FDR control.

ABSTRACT

We propose a new adaptive empirical Bayes framework, the Bag-Of-Null-Statistics (BONuS) procedure, for multiple testing where each hypothesis testing problem is itself multivariate or nonparametric. BONuS is an adaptive and interactive knockoff-type method that helps improve the testing power while controlling the false discovery rate (FDR), and is closely connected to the "counting knockoffs" procedure analyzed in Weinstein et al. (2017). Contrary to procedures that start with a $p$-value for each hypothesis, our method analyzes the entire data set to adaptively estimate an optimal $p$-value transform based on an empirical Bayes model. Despite the extra adaptivity, our method controls FDR in finite samples even if the empirical Bayes model is incorrect or the estimation is poor. An extension, the Double BONuS procedure, validates the empirical Bayes model to guard against power loss due to model misspecification.

Motivation & Objective

• To address the low power of agnostic multivariate tests in multiple testing scenarios where each hypothesis involves high-dimensional or nonparametric data.
• To develop a method that learns an optimal test statistic from the joint data distribution to improve average power across hypotheses.
• To ensure finite-sample FDR control even when the empirical Bayes model is misspecified or poorly estimated.
• To provide a robust, interactive framework that allows analysts to refine test statistics using synthetic controls without violating FDR guarantees.

Proposed method

• BONuS uses a bag-of-null-statistics approach to estimate a data-adaptive p-value transformation by pooling information across all n multivariate hypotheses.
• It applies a masked data scheme where a portion of the data is hidden from the analyst, enabling safe, interactive model fitting while preserving finite-sample FDR control.
• The method estimates an optimal sequence of nested rejection regions by selecting the largest region for which an FDP estimator remains below a pre-specified α level.
• It employs an empirical Bayes model to learn a prior distribution over alternative parameters from the data, focusing on the most relevant directions of deviation from the null.
• The Double BONuS extension uses cross-validation-like validation to test multiple models and select the one with the best performance, guarding against model misspecification.
• For multivariate normal and multinomial settings, BONuS uses robust covariance estimation and synthetic control generation to construct valid null distributions under dependence.

Experimental results

Research questions

• RQ1 Can a data-adaptive test statistic significantly improve power in multivariate multiple testing compared to agnostic methods like GLRT, especially in low-dimensional settings?
• RQ2 Does the BONuS procedure maintain finite-sample FDR control even when the empirical Bayes prior model is incorrect or poorly estimated?
• RQ3 How does BONuS perform in real-world applications such as genome-wide association studies with multivariate phenotypes?
• RQ4 Can the Double BONuS procedure effectively guard against power loss due to model misspecification through model validation?
• RQ5 To what extent can BONuS outperform standard FDR procedures like Benjamini-Hochberg when the underlying signal is sparse and directional?

Key findings

• In a 10-dimensional Gaussian simulation, BONuS with a plug-in estimator of the signal direction achieved nearly oracle-level power, significantly outperforming the GLRT and agnostic methods.
• In multiple testing, BONuS achieved substantially higher true discovery proportions than the Benjamini-Hochberg procedure using GLRT statistics, especially at low α levels.
• In the metabolic syndrome GWAS, BONuS detected more discoveries than agnostic methods, with a measurable but more modest gain due to the expected sparsity of effects per SNP.
• The Double BONuS procedure demonstrated robustness by validating multiple models and selecting the best-performing one, reducing power loss from model misspecification.
• BONuS maintained finite-sample FDR control across all experiments, even when the empirical Bayes model was misspecified, confirming its theoretical robustness.
• The method's performance was stable under dependence structures, as shown in the multinomial and GWAS applications, where robust covariance estimation enabled valid synthetic controls.