[Paper Review] Multiple Test Functions and Adjusted p-Values for Test Statistics with Discrete Distributions

Joshua Habiger|arXiv (Cornell University)|Dec 1, 2014

Statistical Methods in Clinical Trials14 references7 citations

TL;DR

This paper introduces a unified framework for multiple testing with discrete test statistics, proposing adjusted p-values and test functions—particularly the abstract randomized p-value—that reduce bias and variability compared to traditional randomized or nonrandomized approaches. It demonstrates that reporting adjusted abstract randomized p-values and test functions is superior when decisions differ between methods, especially in large-scale multiple testing scenarios.

ABSTRACT

The randomized $p$-value, (nonrandomized) mid-$p$-value and abstract randomized $p$-value have all been recommended for testing a null hypothesis whenever the test statistic has a discrete distribution. This paper provides a unifying framework for these approaches and extends it to the multiple testing setting. In particular, multiplicity adjusted versions of the aforementioned $p$-values and multiple test functions are developed. It is demonstrated that, whenever the usual nonrandomized and randomized decisions to reject or retain the null hypothesis may differ, the (adjusted) abstract randomized $p$-value and test function should be reported, especially when the number of tests is large. It is shown that the proposed approach dominates the traditional randomized and nonrandomized approaches in terms of bias and variability. Tools for plotting adjusted abstract randomized $p$-values and for computing multiple test functions are developed. Examples are used to illustrate the method and to motivate a new type of multiplicity adjusted mid-$p$-value.

Motivation & Objective

• To address the limitations of traditional p-values in multiple testing when test statistics have discrete distributions.
• To unify and extend existing approaches—randomized p-values, mid-p-values, and abstract randomized p-values—into a coherent framework for multiple testing.
• To demonstrate that adjusted abstract randomized p-values and test functions outperform conventional methods in terms of bias and variability.
• To provide practical tools for plotting adjusted abstract randomized p-values and computing multiple test functions in high-dimensional settings.
• To guide practitioners in choosing more informative and statistically sound reporting methods when decisions based on randomized and nonrandomized p-values diverge.

Proposed method

• Develops a unifying framework that treats the test function and abstract randomized p-value as complementary tools for inference under discrete distributions.
• Introduces multiplicity-adjusted versions of the abstract randomized p-value and test function, ensuring strong control of the family-wise error rate.
• Uses the expectation of the test function under the null to define the size of the test, ensuring correct Type I error control.
• Applies the law of iterated expectation to show that the variance of the decision rule is minimized when using the abstract randomized p-value.
• Proposes plotting adjusted abstract randomized p-values as intervals to convey uncertainty, especially when test functions are not 0 or 1.
• Derives and proves that the abstract randomized p-value-based test function dominates both randomized and nonrandomized approaches in terms of statistical efficiency.

Experimental results

Research questions

• RQ1 How can p-values and test functions be adjusted to maintain strong error rate control in multiple testing with discrete test statistics?
• RQ2 When do randomized and nonrandomized decisions differ, and what is the statistical consequence of ignoring such discrepancies?
• RQ3 Can the abstract randomized p-value be extended to multiple testing, and does it dominate traditional approaches in terms of bias and variability?
• RQ4 What is the role of the test function in communicating uncertainty when p-values are close to the significance threshold?
• RQ5 How can practitioners effectively report results when the mid-p-value and randomized p-value lead to conflicting decisions?

Key findings

• The adjusted abstract randomized p-value and test function dominate both randomized and nonrandomized approaches in terms of bias and variability, especially when test functions are not 0 or 1.
• In settings with discrete test statistics, the nonrandomized mid-p-value can lead to incorrect decisions when the p-value is slightly above 0.05, even if the true size is close to 0.05.
• The abstract randomized p-value provides a more accurate representation of the decision uncertainty, as it is defined over an unobserved uniform random variable and maintains exact size control.
• When test functions are not 0 or 1, reporting the adjusted abstract randomized p-value as an interval (e.g., upper and lower bounds) is more informative than relying on a single p-value.
• The proposed method ensures strong family-wise error rate control and is particularly advantageous in large-scale multiple testing where discreteness amplifies decision uncertainty.
• Theoretical proofs confirm that the variance of the decision rule is minimized under the abstract randomized p-value framework, supporting its statistical dominance.