[Paper Review] Graph-Based Tests for Two-Sample Comparisons of Categorical Data

Hao Chen, Nancy R. Zhang|arXiv (Cornell University)|Aug 28, 2012

Bayesian Modeling and Causal Inference12 references4 citations

TL;DR

This paper proposes a graph-based non-parametric method for two-sample comparison of categorical data in high-dimensional, sparse contingency tables. By leveraging category similarity via minimum spanning trees (MST) and minimum distance pairings (MDP), the approach handles tied distances common in categorical data, achieving high power and fast computation with asymptotically normal permutation null distributions and accurate p-value approximations.

ABSTRACT

We study the problem of two-sample comparison with categorical data when the contingency table is sparsely populated. In modern applications, the number of categories is often comparable to the sample size, causing existing methods to have low power. When the number of categories is large, there is often underlying structure on the sample space that can be exploited. We propose a general non-parametric approach that utilizes similarity information on the space of all categories in two sample tests. Our approach extends the graph-based tests of Friedman and Rafsky (1979) and Rosenbaum (2005), which are tests base on graphs connecting observations by similarity. Both tests require uniqueness of the underlying graph and cannot be directly applied on categorical data. We explored different ways to extend graph-based tests to the categorical setting and found two types of statistics that are both powerful and fast to compute. We showed that their permutation null distributions are asymptotically normal and that their $p$-value approximations under typical settings are quite accurate, facilitating the application of the new approach. The approach is illustrated through several examples.

Motivation & Objective

• To address low statistical power in classical chi-square and exact tests when testing two samples with categorical data under sparsity and high dimensionality.
• To exploit underlying similarity structures among categories—such as in rankings, haplotypes, or word frequencies—where traditional methods fail due to sparse contingency tables.
• To extend graph-based tests (Friedman & Rafsky, Rosenbaum) to categorical data by resolving the issue of non-unique graphs caused by tied distances.
• To develop computationally efficient statistics with asymptotically normal null distributions and accurate p-value approximations for practical use.

Proposed method

• Proposes two test statistics: one based on all possible MSTs (aMST) and another on the union of all MSTs (uMST), both using category-level similarity to define edges.
• Uses a sliding transformation algorithm (Eppstein, 1995) to enumerate all MSTs efficiently on the category space, enabling exact computation of the aMST statistic.
• Applies minimum distance pairing (MDP) on categories to define a second test statistic (R_x), with a combinatorial derivation of its null distribution under symmetry.
• Derives the exact null distribution of the MDP-based statistic R_x using combinatorics of perfect matchings, accounting for even/odd group sizes.
• Employs a pseudo-category trick to handle odd total sample sizes by adding a zero-distance subject, later discarding its edge.
• Computes both test statistics in O(K² + M) time, where M is the number of spanning trees, making the method scalable for large K.

Experimental results

Research questions

• RQ1 Can graph-based two-sample tests be effectively extended to categorical data with tied distances and sparse contingency tables?
• RQ2 How can similarity information among categories be leveraged to improve statistical power in high-dimensional categorical testing?
• RQ3 What is the exact null distribution of graph-based test statistics when multiple MSTs or MDPs exist due to ties?
• RQ4 Can p-value approximations for these statistics be accurate enough to avoid full permutation testing in practice?
• RQ5 How does the computational complexity scale with the number of categories, and can efficient algorithms be developed for real-world applications?

Key findings

• The proposed aMST and uMST statistics are both powerful and computationally efficient, with time complexity O(K² + M), where M is the number of spanning trees.
• The permutation null distribution of both test statistics is asymptotically normal, enabling accurate p-value approximation without full permutation resampling.
• The exact null distribution of the MDP-based statistic R_x is derived combinatorially, with closed-form expressions for the number of pairings with i cross-group edges.
• The p-value approximations for both test statistics are accurate under typical settings, reducing reliance on computationally intensive permutations.
• The method successfully handles tied distances in categorical data by using unions of MSTs (uMST) and all possible MSTs (aMST), overcoming a key limitation of prior graph-based tests.
• The approach is validated on real-world problems such as preference rankings, haplotype association, and document comparison, where classical tests fail due to sparsity.