[Paper Review] Two-sample Testing for Large, Sparse High-Dimensional Multinomials under Rare/Weak Perturbations

David L. Donoho, Alon Kipnis|arXiv (Cornell University)|Jul 3, 2020

Statistical Methods and Inference22 references4 citations

TL;DR

This paper develops a Higher Criticism-based test for two-sample homogeneity in large, sparse high-dimensional multinomials under rare/weak perturbations. By combining P-values from exact binomial tests across N categories, it identifies a phase transition in the (β,r)-plane where the test achieves asymptotic maximality in power, with performance matching a normal means model in dense settings.

ABSTRACT

Given two samples from possibly different discrete distributions over a common set of size $N$, consider the problem of testing whether these distributions are identical, vs. the following rare/weak perturbation alternative: the frequencies of $N^{1-\beta}$ elements are perturbed by $r(\log N)/2n$ in the Hellinger distance, where $n$ is the size of each sample. We adapt the Higher Criticism (HC) test to this setting using P-values obtained from $N$ exact binomial tests. We characterize the asymptotic performance of the HC-based test in terms of the sparsity parameter $\beta$ and the perturbation intensity parameter $r$. Specifically, we derive a region in the $(\beta,r)$-plane where the test asymptotically has maximal power, while having asymptotically no power outside this region. Our analysis distinguishes between the cases of dense ($N\gg n$) and sparse ($N\ll n$) contingency tables. In the dense case, the phase transition curve matches that of an analogous two-sample normal means model.

Motivation & Objective

• To address two-sample homogeneity testing in high-dimensional discrete distributions where only a small fraction of categories are perturbed.
• To model perturbations as rare (N^{1−β} elements) and weak (r(log N)/(2n) in Hellinger distance), reflecting realistic sparse signal scenarios.
• To adapt the Higher Criticism test to this multinomial setting using P-values from exact binomial tests on each category.
• To characterize the asymptotic detection boundary in terms of sparsity β and perturbation intensity r.
• To compare performance across dense (N ≫ n) and sparse (N ≪ n) contingency table regimes.

Proposed method

• The method computes exact binomial P-values for each of the N categories under the null of identical distributions across two samples of size n.
• These P-values are aggregated using the Higher Criticism (HC) statistic to detect deviations from the uniform null distribution.
• The test statistic is derived under the assumption that only N^{1−β} categories are perturbed, with Hellinger distance perturbations of size r(log N)/(2n).
• Asymptotic power analysis is conducted in the limit as N → ∞, with n growing accordingly, distinguishing between dense (N ≫ n) and sparse (N ≪ n) regimes.
• The phase transition curve is derived by analyzing the behavior of the HC test statistic under the alternative, identifying the boundary between detectable and undetectable perturbations.
• The analysis leverages extreme value theory and tail behavior of P-values to characterize detection limits.

Experimental results

Research questions

• RQ1 In what region of the (β,r)-plane is the Higher Criticism test asymptotically powerful for detecting rare/weak perturbations in high-dimensional multinomials?
• RQ2 How does the performance of the HC-based test compare between dense and sparse contingency tables in this setting?
• RQ3 Does the detection boundary for the multinomial two-sample test match that of the analogous two-sample normal means model in the dense regime?
• RQ4 What is the role of exact binomial P-values in enabling detection under weak and sparse alternatives?
• RQ5 How does the sparsity parameter β influence the detectability of perturbations in high-dimensional discrete data?

Key findings

• The HC-based test achieves asymptotic maximality in power within a specific region of the (β,r)-plane, defined by the phase transition curve.
• Outside this region, the test has asymptotically no power, indicating a sharp detection boundary.
• In the dense regime (N ≫ n), the phase transition curve for the multinomial test matches that of the two-sample normal means model.
• The method remains effective in the sparse regime (N ≪ n), though the detection boundary differs from the dense case.
• The use of exact binomial P-values enables sensitivity to weak, sparse perturbations that would be missed by standard chi-squared tests.
• The analysis confirms that the Higher Criticism approach is optimal in this high-dimensional, sparse, rare/weak setting.