[Paper Review] True and false discoveries with independent and sequential e-values

Vladimir Vovk, Ruodu Wang|arXiv (Cornell University)|Mar 1, 2020

Statistical Methods in Clinical Trials6 references4 citations

TL;DR

This paper proposes a novel discovery matrix algorithm using independent e-values and the U₂ e-merging function (sum of pairwise products), which significantly improves false discovery control and statistical power over existing methods based on p-values or the arithmetic mean (U₁). The method leverages the relative variance of e-values to enhance detection of true discoveries while maintaining strong frequentist error control, outperforming prior approaches in simulations and theoretical analysis.

ABSTRACT

In this paper we use e-values in the context of multiple hypothesis testing assuming that the base tests produce independent, or sequential, e-values. Our simulation and empirical studies and theoretical considerations suggest that, under this assumption, our new algorithms are superior to the known algorithms using independent p-values and to our recent algorithms designed for e-values without the assumption of independence.

Motivation & Objective

• To develop a more powerful and reliable method for multiple hypothesis testing under independence of e-values.
• To improve upon existing e-value-based discovery procedures that rely on the arithmetic mean (U₁) or p-values.
• To design a computationally efficient discovery matrix algorithm that maintains strong error control while maximizing true discovery detection.
• To theoretically and empirically demonstrate the superiority of U₂ over U₁ and p-value-based methods in controlling false discoveries.

Proposed method

• The method uses the U₂ e-merging function, defined as the average of all pairwise products of e-values, to aggregate evidence across independent tests.
• A discovery matrix is constructed via Algorithm 1, which computes thresholds D_{r,j} based on the minimum U₂ value over all r-subsets of the top j hypotheses.
• The algorithm ensures validity under any underlying probability measure by relying on the e-variable property of U₂ under independence.
• The method is calibrated to a color-coded evidence scale (green to black) based on U₂ values, reflecting strength of evidence for at least j true discoveries among r rejections.
• The approach generalizes the e-value framework by exploiting the variance structure of e-values through the relative variance rvar(e).
• An e-to-p calibration is applied, transforming e-values to p-values for comparison with classical methods like GWGS.

Experimental results

Research questions

• RQ1 Can a discovery matrix based on U₂ e-values outperform existing methods in terms of true discovery detection and false discovery control?
• RQ2 How does the performance of U₂ compare to U₁ (arithmetic mean) and p-value-based procedures under independent e-values?
• RQ3 What is the theoretical relationship between the relative variance of e-values and the performance of U₂-based discovery matrices?
• RQ4 Can the U₂-based method achieve stronger evidence for true discoveries while maintaining strong error control?

Key findings

• The U₂-based discovery matrix outperforms both U₁-based and p-value-based methods in simulation studies, particularly in detecting true discoveries with stronger evidence.
• The method achieves a squared improvement in discovery matrix entries when e-values are identical, due to the U₂ formula U₂(e) = M₁²(1 − rvar(e)).
• The relative variance rvar(e) quantifies e-value diversity, and performance degrades as rvar(e) increases, but the method remains robust under independence.
• After e-to-p calibration, the U₂-based p-values show stronger evidence (e.g., more black/dark red entries) than the GWGS procedure designed for p-values.
• The algorithm runs in O(K⁵) time for U₂, which is computationally feasible and significantly faster than naive O(K⁶) approaches.
• Theoretical analysis confirms that U₂ is a valid ie-merging function under independence and dominates U₁ in detecting true discoveries when e-values are heterogeneous.