[Paper Review] Interpretable hypothesis tests

Victor Coscrato, Luís Gustavo Esteves+2 more|arXiv (Cornell University)|Apr 13, 2019

Meta-analysis and systematic reviews23 references4 citations

TL;DR

This paper proposes interpretable hypothesis tests by integrating agnostic tests—allowing three outcomes: reject, accept, or remain agnostic—and pragmatic hypotheses that replace precise nulls with effect-size-based alternatives. By controlling both Type I and II errors and ensuring logical consistency in multiple testing, the approach resolves interpretive challenges in standard tests, particularly regarding non-rejection ambiguity and practical relevance of rejections.

ABSTRACT

Although hypothesis tests play a prominent role in Science, their interpretation can be challenging. Three issues are (i) the difficulty in making an assertive decision based on the output of an hypothesis test, (ii) the logical contradictions that occur in multiple hypothesis testing, and (iii) the possible lack of practical importance when rejecting a precise hypothesis. These issues can be addressed through the use of agnostic tests and pragmatic hypotheses.

Motivation & Objective

• Address the interpretive difficulty of standard hypothesis tests, where non-rejection of the null is ambiguous and often misinterpreted as evidence in favor of the null.
• Resolve logical inconsistencies in multiple hypothesis testing, such as rejecting a hypothesis while not rejecting a logically implied one.
• Ensure that statistical rejections have practical relevance by replacing precise hypotheses with pragmatic alternatives based on meaningful effect sizes.
• Provide a framework that enables clearer decision-making in statistical inference through explicit control of error types and interpretable outcomes.

Proposed method

• Implement agnostic hypothesis tests that allow three outcomes: reject the null, accept the null, or remain agnostic, thereby avoiding the 'limbo of suspended disbelief' in non-rejection.
• Control both Type I and Type II error rates by explicitly modeling the 'agnostic' outcome, which is known when it occurs, unlike in standard tests.
• Define pragmatic hypotheses as sets of parameter values that produce similar predictive behavior to the null, such as $Pg(H_0) = \{\theta^* : d_Z(\theta_0, \theta^*) \leq \epsilon\}$, to ensure practical relevance.
• Use Monte Carlo integration to approximate pragmatic hypotheses in high-dimensional models when exact computation is infeasible, fixing nuisance parameters at their estimates.
• Apply region-based agnostic tests to pragmatic hypotheses, enabling decisions that reflect both statistical and practical significance.
• Construct confidence regions for parameters under pragmatic hypotheses and assess test outcomes across varying sample sizes to demonstrate stability and interpretability.

Experimental results

Research questions

• RQ1 How can hypothesis tests be restructured to avoid the interpretive ambiguity of non-rejection in standard tests?
• RQ2 Can logical inconsistencies in multiple hypothesis testing be eliminated through a modified testing framework?
• RQ3 To what extent does replacing precise null hypotheses with pragmatic alternatives improve the practical relevance of statistical rejections?
• RQ4 How can agnostic tests be constructed to control both Type I and Type II error rates while allowing a third, neutral outcome?
• RQ5 What is a feasible and general method for approximating pragmatic hypotheses in parametric models with multiple parameters?

Key findings

• Agnostic tests eliminate the 'limbo of suspended disbelief' by explicitly distinguishing between lack of evidence and evidence in favor of the null hypothesis.
• The inclusion of an 'agnostic' outcome allows control over both Type I and Type II errors, and the Type III error (remaining agnostic) is known and manageable, unlike in standard tests.
• Agnostic tests guarantee logically coherent conclusions in multiple hypothesis testing, resolving contradictions such as rejecting $H_0$ but not rejecting $H_0^*$ when $H_0^*$ implies $H_0$.
• Pragmatic hypotheses, defined as sets of parameter values that induce similar predictions, ensure that a rejection of the null is practically meaningful, not just statistically significant.
• Approximate pragmatic hypotheses can be efficiently computed using Monte Carlo integration by conditioning on estimated nuisance parameters, reducing computational complexity from high-dimensional to one-dimensional search.
• In the gravitational constant example, the test correctly accepts that $g \approx 9.8$ as practically equivalent to the true value $g \approx 9.807$ when measurement imprecision is high, demonstrating practical interpretability across sample sizes.