[Paper Review] Permutation tests of non-exchangeable null models

Jeffrey Roach, William Valdar|arXiv (Cornell University)|Aug 30, 2018

Diverse Scientific and Engineering Research14 references3 citations

TL;DR

This paper generalizes permutation tests to non-exchangeable null models by introducing a framework that maintains exactness and derives the most powerful generalized permutation test in closed form. It establishes convergence of Monte Carlo approximations using Bernstein polynomial-based estimators and provides rigorous bounds on p-value estimation error, enabling practical application in complex models like linear mixed models.

ABSTRACT

Generalizations to the permutation test are introduced to allow for situations in which the null model is not exchangeable. It is shown that the generalized permutation tests are exact, and a partial converse: that any test function that is exact on all probability densities coincides with a generalized permutation test on a particular region, is established. A most powerful generalized permutation test is derived in closed form. Approximations to the most powerful generalized permutation test are proposed to reduce the computational burden required to compute the complete test. In particular, an explicit form for the approximate test is derived in terms of a multinomial Bernstein polynomial approximation, and its convergence to the most powerful generalized permutation test is demonstrated. In the case where the determination of p-values is of greater interest than testing of hypotheses, two approaches to estimation of significance are analyzed. Bounds on the deviation from significance of the exact most powerful test are given in terms of sample size. For both estimators, as sample size approaches infinity, the estimator converges to the significance of the most powerful generalized permutation test under mild conditions. Applications of generalized permutation testing to linear mixed models are provided.

Motivation & Objective

• To extend permutation testing beyond exchangeable null models, which are often unrealistic in practice due to outlier sensitivity and structural constraints.
• To develop a generalized permutation test that remains exact under non-exchangeable nulls, ensuring valid Type I error control.
• To derive the most powerful generalized permutation test in closed form through linear optimization, linking it to Neyman-Pearson theory.
• To provide computationally feasible approximations using Monte Carlo sampling and multinomial Bernstein polynomial estimators.
• To analyze p-value estimation accuracy, offering probabilistic bounds on deviation from the exact most powerful test.

Proposed method

• Generalized permutation tests are defined via a probability measure on permutations that reflects the non-exchangeable null density $ g_0 $, reducing to standard permutation tests when $ g_0 $ is exchangeable.
• The most powerful test is derived by solving a canonical linear programming problem over permutation space, maximizing power under the null distribution.
• A Monte Carlo approximation is constructed by sampling permutations according to an estimated density $ \hat{g}_0 $, with selection probabilities defined by integrals over asymmetric units.
• The estimator $ \Sigma_U $, based on the fraction of permutations with likelihood ratio exceeding a threshold, is shown to be unbiased for the true p-value.
• Bernstein polynomial approximations are used to model the distribution of test statistics, enabling convergence to the exact most powerful test as sample size increases.
• Hoeffding’s and Bernstein’s inequalities are applied to bound the deviation of the p-value estimator from the true significance level, ensuring probabilistic convergence.

Experimental results

Research questions

• RQ1 Can permutation tests be generalized to non-exchangeable null models while preserving exactness and power?
• RQ2 What is the form of the most powerful generalized permutation test under a non-exchangeable null model?
• RQ3 How can the most powerful test be approximated efficiently without computing all $ n! $ permutations?
• RQ4 What are the convergence properties and error bounds of Monte Carlo p-value estimators in this generalized framework?
• RQ5 How do approximation schemes based on Bernstein polynomials compare to direct Monte Carlo sampling in terms of accuracy and efficiency?

Key findings

• The generalized permutation test is exact under any non-exchangeable null model, and any exact test under all probability densities coincides with such a generalized test on a large subset of the sample space.
• The most powerful generalized permutation test is derived in closed form by reducing the problem to a linear optimization task over the permutation group.
• The Monte Carlo approximation based on sampling permutations according to $ \hat{g}_0 $ converges almost surely to the true most powerful test as the number of sampled permutations increases.
• The p-value estimator $ \Sigma_U $, based on the fraction of permutations with likelihood ratio exceeding $ l(\mathbf{x}) $, is unbiased for the true significance level.
• Hoeffding’s and Bernstein’s inequalities provide explicit probabilistic bounds on the deviation of the p-value estimator from the true value, with error decreasing exponentially in the number of samples.
• The multinomial Bernstein polynomial approximation provides a smooth, convergent estimator of the most powerful test, with convergence established under mild regularity conditions on $ \hat{g}_0 $.