[Paper Review] Control of Generalized Error Rates in Multiple Testing

Joseph P. Romano, Michael Wolf|Zurich Open Repository and Archive (University of Zurich)|May 1, 2005

Statistical Methods in Clinical Trials30 references106 citations

TL;DR

This paper proposes resampling-based procedures to control generalized error rates in multiple hypothesis testing, including the k-FWER (probability of at least k false rejections) and the FDP (false discovery proportion). Using bootstrap and subsampling methods, the authors develop step-down procedures that account for dependence among test statistics, achieving asymptotic control of these error rates without requiring subset pivotality, thus improving power in high-dimensional settings like genomics.

ABSTRACT

Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $γ$ and $α$, $P\{FDP>γ\}\le α$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Motivation & Objective

• To develop computationally feasible procedures that control generalized error rates in multiple testing under weak dependence assumptions.
• To extend beyond traditional familywise error rate (FWER) control by allowing controlled levels of false rejections to improve statistical power.
• To address the limitations of existing methods by incorporating dependence structures among test statistics through resampling.
• To provide asymptotic control of the k-FWER and the probability that the false discovery proportion exceeds a threshold γ.
• To offer a practical alternative to FDR and FWER control that balances Type I and Type II error trade-offs in high-dimensional inference.

Proposed method

• Uses bootstrap and subsampling to estimate the joint distribution of test statistics under the null, enabling accurate critical value calculation.
• Employs the k-max statistic to identify the kth largest test statistic among true nulls, forming the basis for k-FWER control.
• Applies a step-down procedure that sequentially rejects hypotheses based on ordered test statistics and resampled critical values.
• Introduces a novel algorithm for FDP control that dynamically adjusts the rejection threshold based on bootstrap estimates of false discovery proportion.
• Relies on asymptotic theory and U-statistic arguments to justify the validity of the resampling-based critical values.
• Avoids the subset pivotality assumption required by some other methods, enhancing applicability to dependent test statistics.

Experimental results

Research questions

• RQ1 Can resampling-based procedures control the k-FWER in finite samples or asymptotically under weak dependence assumptions?
• RQ2 Can the false discovery proportion (FDP) be controlled such that P(FDP > γ) ≤ α asymptotically, for a user-specified γ ∈ [0,1)?
• RQ3 How can dependence among test statistics be leveraged to improve the power of multiple testing procedures without violating error rate control?
• RQ4 Do the proposed step-down procedures outperform existing single-step or FDR-controlling methods in terms of power and error rate control?
• RQ5 Is the proposed method robust to dependence structures that violate the subset pivotality condition?

Key findings

• The proposed step-down procedures achieve asymptotic control of the k-FWER under weak regularity conditions, without requiring the subset pivotality assumption.
• The method controls P(FDP > γ) ≤ α asymptotically for any γ ∈ [0,1), providing a more flexible alternative to FDR control.
• Simulations show improved power compared to existing methods, particularly when test statistics are dependent.
• The k-FWER procedure is monotonic in k: increasing k leads to more rejections, which supports the validity of the FDP control algorithm.
• Theoretical justification relies on subsampling and U-statistic theory, ensuring asymptotic validity under general dependence.
• A counterexample in the appendix shows that a related method from [33] fails to control FDP even asymptotically when all nulls are true, highlighting the novelty and necessity of the proposed approach.