[Paper Review] The Power of Batching in Multiple Hypothesis Testing

Tijana Zrnic, Daniel Jiang+2 more|arXiv (Cornell University)|Oct 11, 2019

Statistical Methods in Clinical Trials21 references4 citations

TL;DR

This paper introduces Batch_BH and Batch_St-BH, novel FDR-controlling algorithms that enable sequential, adaptive hypothesis testing across multiple batches while preserving formal FDR control. By reapplying the Benjamini-Hochberg and Storey-BH procedures within a batched framework, the methods achieve higher statistical power than online methods and stronger FDR guarantees than naive batching, bridging the gap between offline and online multiple testing approaches.

ABSTRACT

One important partition of algorithms for controlling the false discovery rate (FDR) in multiple testing is into offline and online algorithms. The first generally achieve significantly higher power of discovery, while the latter allow making decisions sequentially as well as adaptively formulating hypotheses based on past observations. Using existing methodology, it is unclear how one could trade off the benefits of these two broad families of algorithms, all the while preserving their formal FDR guarantees. To this end, we introduce $ ext{Batch}_{ ext{BH}}$ and $ ext{Batch}_{ ext{St-BH}}$, algorithms for controlling the FDR when a possibly infinite sequence of batches of hypotheses is tested by repeated application of one of the most widely used offline algorithms, the Benjamini-Hochberg (BH) method or Storey's improvement of the BH method. We show that our algorithms interpolate between existing online and offline methodology, thus trading off the best of both worlds.

Motivation & Objective

• To address the trade-off between the high statistical power of offline FDR methods and the adaptivity of online FDR methods.
• To develop a framework for controlling the false discovery rate when testing an infinite sequence of hypothesis batches in an online manner.
• To preserve formal FDR control when repeatedly applying offline FDR procedures (e.g., BH and Storey-BH) across batches, which otherwise leads to uncontrolled FDR.
• To enable adaptive hypothesis generation based on past results while maintaining FDR control over both the full sequence and individual batches.
• To provide a method that inherits the stability of offline methods and the sequential decision-making capability of online methods.

Proposed method

• Proposes Batch_BH and Batch_St-BH as sequential compositions of the Benjamini-Hochberg and Storey-BH procedures across multiple batches.
• Applies the BH and Storey-BH procedures independently to each batch, using a diminishing significance level per batch to maintain overall FDR control.
• Uses a conservative dependence assumption on the p-value sequence to ensure FDR control under general conditions.
• Derives theoretical FDR bounds using conditional expectation and tower property, showing that FDR is controlled up to a factor of max{1, δ} times the nominal level α.
• Introduces a correction term ε_i to account for potential imbalance in expected discovery counts depending on whether a null hypothesis is rejected.
• Extends the framework to handle positively dependent p-values through a modified batched procedure with adjusted significance levels.

Experimental results

Research questions

• RQ1 Can we design a method that combines the high power of offline FDR procedures with the adaptivity of online procedures in sequential hypothesis testing?
• RQ2 Is it possible to repeatedly apply offline FDR algorithms (like BH or Storey-BH) across batches without violating FDR control guarantees?
• RQ3 How can we ensure FDR control not only over the entire sequence of tests but also over individual batches or subgroups?
• RQ4 What theoretical bounds can be established for the FDR of a batched procedure that reuses offline algorithms in an online setting?
• RQ5 How does the performance of the proposed batched methods compare to purely online or purely offline FDR methods in terms of statistical power and FDR control?

Key findings

• Batch_BH and Batch_St-BH achieve formal FDR control over both the entire sequence of tests and each individual batch, unlike naive sequential application of offline methods.
• The FDR of the proposed methods is bounded by max{1, δ} × α, where δ is the supremum of the ratio of expected discoveries given a null hypothesis is rejected versus not rejected.
• The methods inherit the stability of offline procedures, as their discovery sets are invariant to the ordering of hypotheses, unlike online methods.
• Empirical results on synthetic data show that Batch_BH and Batch_St-BH achieve higher statistical power than state-of-the-art online FDR methods while maintaining FDR control.
• The framework allows for adaptive hypothesis generation based on past results, enabling applications in large-scale, sequential testing such as A/B testing in industry.
• The algorithms have been integrated into the onlineFDR R package, demonstrating practical usability and adoption.