[Paper Review] Estimating the number and effect sizes of non-null hypotheses

Jennifer Brennan, Ramya Korlakai Vinayak+1 more|arXiv (Cornell University)|Feb 17, 2020

Statistical Methods in Clinical Trials36 references4 citations

TL;DR

This paper proposes a computationally efficient, conservative estimator for the number and effect sizes of non-null hypotheses in multiple testing settings using a minimal pilot experiment. By leveraging an $ε$-ball around the empirical CDF and minimizing mass above threshold $\gamma$, the method guarantees conservative estimates of discoveries with high probability, enabling optimal experimental design with significantly fewer samples than full-scale experiments.

ABSTRACT

We study the problem of estimating the distribution of effect sizes (the mean of the test statistic under the alternate hypothesis) in a multiple testing setting. Knowing this distribution allows us to calculate the power (type II error) of any experimental design. We show that it is possible to estimate this distribution using an inexpensive pilot experiment, which takes significantly fewer samples than would be required by an experiment that identified the discoveries. Our estimator can be used to guarantee the number of discoveries that will be made using a given experimental design in a future experiment. We prove that this simple and computationally efficient estimator enjoys a number of favorable theoretical properties, and demonstrate its effectiveness on data from a gene knockout experiment on influenza inhibition in Drosophila.

Motivation & Objective

• To address the challenge of designing experiments with guaranteed statistical power when effect sizes are unknown.
• To estimate the distribution of effect sizes across multiple hypotheses using a small, inexpensive pilot study.
• To provide a conservative estimator that never overestimates the number of discoveries above a given effect size threshold.
• To enable experimental design trade-offs between cost (replicates) and discovery yield using theoretical guarantees.
• To develop a method that is both computationally efficient and statistically robust for real-world applications like gene knockout studies.

Proposed method

• The estimator constructs an $\ell_\infty$ ball around the empirical cumulative distribution function (CDF) of test statistics to define a set of plausible distributions.
• It finds the distribution within this ball that minimizes the probability mass above a given threshold $\gamma$, ensuring conservative estimates.
• The method uses binary search over the unit interval to compute the lower-bound estimate $\widehat{\zeta}_n(\gamma)$, with hypothesis testing at each step to validate mass thresholds.
• The algorithm relies on convex optimization (via CVXPY and ECOS) to solve the constrained minimization problem efficiently.
• The estimator is designed to satisfy $\widehat{\zeta}_n(\gamma) \leq \zeta_{\nu_*}(\gamma)$ with high probability, ensuring no overestimation of discoveries.
• The approach is applied to Gaussian, Poisson, and binomial test statistics, with validation on real Drosophila influenza gene knockout data.

Experimental results

Research questions

• RQ1 Can we estimate the number of non-null hypotheses with effect sizes above a threshold $\gamma$ using significantly fewer samples than required for full discovery?
• RQ2 How can we design a conservative estimator that never overestimates the number of discoveries while remaining computationally efficient?
• RQ3 What theoretical guarantees can be provided for the estimator’s accuracy in finite samples?
• RQ4 How does the estimator compare to plug-in and other baseline methods in terms of conservativeness and accuracy?
• RQ5 Can the estimator guide experimental design by balancing cost (replicates) and discovery yield?

Key findings

• The proposed estimator provides finite-sample upper and lower bounds on the estimated fraction of hypotheses above threshold $\gamma$, ensuring statistical reliability.
• The estimator is provably conservative: $\mathbb{P}(\widehat{\zeta}_n(\gamma) > \zeta_{\nu_*}(\gamma)) \to 0$ as sample size increases, guaranteeing no overestimation of discoveries.
• The method achieves better accuracy than baseline estimators as sample size $n$ increases, while baselines do not improve with more data.
• In simulations with $n=100,000$ samples, the estimator closely tracks the true $\zeta_*$ without overestimation, even under low signal-to-noise conditions.
• Theoretical analysis shows that any estimator achieving accuracy $\varepsilon$ over a range $A_\varepsilon$ requires at least $n \gtrsim 1/(\varepsilon^2 \gamma_*^4)$ samples, establishing a lower bound on sample complexity.
• The method was successfully applied to real data from a Drosophila gene knockout experiment, demonstrating practical utility in experimental design.