[Paper Review] Hypothesis testing with e-values

Aaditya Ramdas, Ruodu Wang|arXiv (Cornell University)|Oct 31, 2024

Advanced Statistical Process Monitoring5 citations

TL;DR

This paper surveys e-values as a fundamental tool for hypothesis testing, detailing their definitions, properties, and relations to p-values, and presents methods for constructing, aggregating, and applying e-values in sequential, multiple testing, and risk assessment contexts.

ABSTRACT

This book is written to offer a humble, but unified, treatment of e-values in hypothesis testing. It is organized into three parts: Fundamental Concepts, Core Ideas, and Advanced Topics. The first part includes four chapters that introduce the basic concepts. The second part includes five chapters of core ideas such as universal inference, log-optimality, e-processes, operations on e-values, and e-values in multiple testing. The third part contains seven chapters of advanced topics. The book collates important results from a variety of modern papers on e-values and related concepts, and also contains many results not published elsewhere. It offers a coherent and comprehensive picture on a fast-growing research area, and is ready to use as the basis of a graduate course in statistics and related fields.

Motivation & Objective

• Motivate e-values as a foundational tool for hypothesis testing and understanding their three roles: methodological, technical, and fundamental.
• Define e-values, p-values, and tests, and contrast their properties and interpretations with p-values.
• Explain calibration between e-values and p-values and establish when powered e-values imply powered tests.
• Introduce universal inference and log-optimal e-values as core constructs for composite and irregular testing problems.
• Discuss the impact of e-values on sequential, multiple testing, and risk-measure forecasting contexts.

Proposed method

• Definition and interpretation of e-values as nonnegative statistics with expectation at most one under the null.
• Calibration between e-values and p-values to derive valid tests and procedures.
• Construction of universal inference e-values for irregular testing problems and the log-optimal e-value (numeraire).
• Development of e-processes and sequential anytime-valid inference for sequential data.
• Connection of e-values to false discovery rate control and confidence interval procedures via compound e-values and e-BY/e-BH frameworks.
• Exposition of relationships between e-values and Bayes factors, reverse information projection, and martingale methods.

Experimental results

Research questions

• RQ1 What are e-values and how do they relate to p-values under the null and alternative hypotheses?
• RQ2 How can e-values be calibrated to p-values and used to construct valid tests, including in sequential and adaptive settings?
• RQ3 What constitutes a universal or log-optimal e-value, and how do these guarantee power for composite or irregular hypotheses?
• RQ4 How can e-values be aggregated or merged across dependent tests, and how do they facilitate multiple testing and FDR control?
• RQ5 What are the roles of e-values in sequential inference, optional stopping, and risk-measure testing?

Key findings

• E-values provide a unified framework for hypothesis testing that yields valid tests via simple thresholding (e.g., E > 1/α).
• Calibrators exist to transform between e-values and p-values, enabling cross-interpretation while preserving error guarantees.
• Universal inference yields e-values for irregular problems, and a log-optimal e-value always exists for a fixed alternative.
• E-processes and sequential e-values enable anytime-valid inference and avoid issues from peeking or optional stopping.
• Compound e-values underpin false discovery rate control and confidence interval combination, enabling robust multiple testing and FDR procedures.
• Merging and averaging e-values under dependence retain validity, supporting sequential experimentation and risk assessment applications.