[Paper Review] Distribution-free Multiple Testing

Ery Arias-Castro, Shiyun Chen|arXiv (Cornell University)|Apr 26, 2016

Statistical Methods in Clinical Trials19 references6 citations

TL;DR

This paper establishes asymptotic optimality of the Benjamini-Hochberg (BH) procedure and a distribution-free method by Foygel-Barber and Candffs in location models with symmetric null distributions. It proves both methods achieve optimal false discovery rate and false non-discovery rate trade-offs in normal means and generalized Gaussian models, even when the null distribution is unknown, relying only on symmetry and tail decay properties.

ABSTRACT

We study a stylized multiple testing problem where the test statistics are independent and assumed to have the same distribution under their respective null hypotheses. We first show that, in the normal means model where the test statistics are normal Z-scores, the well-known method of (Benjamini and Hochberg, 1995) is optimal in some asymptotic sense. We then show that this is also the case of a recent distribution-free method proposed by Foygel-Barber and Candès (2015). The method is distribution-free in the sense that it is agnostic to the null distribution - it only requires that the null distribution be symmetric. We extend these optimality results to other location models with a base distribution having fast-decaying tails.

Motivation & Objective

• To establish asymptotic optimality of the Benjamini-Hochberg (BH) procedure in the normal means model under known null distributions.
• To extend optimality results to distribution-free multiple testing methods that only assume symmetry of the null distribution.
• To analyze the performance of the Foygel-Barber and Candffs (2015) distribution-free method in location models with fast-decaying tails.
• To show that both BH and the distribution-free method achieve the same first-order asymptotic performance in terms of false discovery and non-discovery rates.
• To unify theoretical understanding of multiple testing under minimal distributional assumptions, particularly symmetry and tail decay.

Proposed method

• Uses a stylized multiple testing framework with independent test statistics and a common null distribution symmetric about zero.
• Defines risk as the sum of false discovery rate (FDR) and false non-discovery rate (FNR), treating this as a minimax risk criterion.
• Applies threshold-based procedures where rejection occurs when test statistics exceed a data-dependent threshold derived from empirical survival functions.
• Employs asymptotic analysis in generalized Gaussian models where the tail decay rate is characterized by a parameter γ, allowing generalization beyond normality.
• Uses empirical process theory and concentration inequalities to control estimation error in the empirical distribution function of test statistics.
• Compares the distribution-free method to BH by showing both achieve the same asymptotic threshold and error rates under symmetry and tail decay conditions.

Experimental results

Research questions

• RQ1 Is the Benjamini-Hochberg procedure asymptotically optimal in the normal means model when the null distribution is known?
• RQ2 Can a distribution-free multiple testing method that only assumes symmetry of the null distribution achieve the same asymptotic performance as BH?
• RQ3 How does the performance of the Foygel-Barber and Candffs (2015) distribution-free method scale in models with heavy-tailed or generalized Gaussian null distributions?
• RQ4 What is the asymptotic behavior of the false discovery and non-discovery rates under minimal assumptions beyond normality?
• RQ5 Can the same thresholding strategy achieve optimal risk (FDR + FNR) in both known and unknown null distribution settings?

Key findings

• The Benjamini-Hochberg procedure is asymptotically optimal to first order in the normal means model, achieving the minimal possible risk (FDR + FNR) as n → ∞.
• The distribution-free method of Foygel-Barber and Candffs (2015), which requires only symmetry of the null distribution, achieves the same asymptotic performance as BH in the normal model.
• In generalized Gaussian models with tail decay rate γ > 1, both BH and the distribution-free method achieve FDR and FNR that converge to zero under appropriate signal strength and sparsity conditions.
• The threshold τ used in the distribution-free method satisfies τ ≤ t* with probability tending to 1, where t* is chosen such that Ψ(t*) ∼ n^{-r*} for r* ∈ (β, r), ensuring sufficient separation from the null.
• The false non-discovery rate of the distribution-free method converges to zero in probability, implying high power to detect true signals under the same conditions as BH.
• The results hold under minimal assumptions: symmetry of the null distribution and fast-decaying tails, making the method robust to misspecification of the null distribution.