[Paper Review] The Classification Permutation Test: A Nonparametric Test for Equality of Multivariate Distributions

Johann A. Gagnon-Bartsch, Yotam Shem-Tov|arXiv (Cornell University)|Nov 19, 2016

Advanced Causal Inference Techniques10 references3 citations

TL;DR

The Classification Permutation Test (CPT) is a nonparametric method that tests equality of multivariate distributions by combining classification algorithms (e.g., logistic regression, random forests) with permutation inference to assess whether covariates can predict treatment assignment. It provides exact finite-sample inference and demonstrates superior power in detecting covariate imbalance compared to existing nonparametric tests, especially when imbalances exist in joint distributions rather than marginal moments.

ABSTRACT

The gold standard for identifying causal relationships is a randomized controlled experiment. In many applications in the social sciences and medicine, the researcher does not control the assignment mechanism and instead may rely upon natural experiments, regression discontinuity designs, RCTs with attrition, or matching methods as a substitute to experimental randomization. The standard testable implication of random assignment is covariate balance between the treated and control units. Covariate balance is therefore commonly used to validate the claim of "as-if" random assignment. We develop a new nonparametric test of covariate balance. Our Classification Permutation Test (CPT) is based on a combination of classification methods (e.g. logistic regression or random forests) with Fisherian permutation inference. The CPT is guaranteed to have correct coverage and is consistent under weak assumptions on the chosen classifier. To illustrate the gains of using the CPT, we revisit four real data examples: Lyall (2009); Green and Winik (2010); Eggers and Hainmueller (2009); and Rouse (1995). Monte Carlo power simulations are used to compare the CPT to two existing nonparametric tests of equality of multivariate distributions.

Motivation & Objective

• To develop a robust, nonparametric test for equality of multivariate distributions in observational studies where random assignment is not controlled.
• To address limitations of existing balance tests that focus only on marginal distributions or mean differences, which may miss complex imbalances in joint covariate distributions.
• To demonstrate how machine learning classifiers can be integrated with rigorous statistical inference to improve detection of covariate imbalance.
• To provide a practical, interpretable, and powerful alternative to balance tables and standard t-tests in applied research across social sciences and biostatistics.

Proposed method

• The CPT trains a classifier (e.g., logistic regression, random forests) to distinguish treated from control units based on observed covariates.
• It computes an out-of-sample classification accuracy rate as the test statistic, measuring how well covariates predict treatment status.
• Permutation inference is used to generate the null distribution: treatment labels are randomly permuted across units, and the classification accuracy is recomputed under each permutation.
• The p-value is calculated as the proportion of permuted samples where the classifier accuracy is at least as high as the observed accuracy, ensuring exact Type I error control.
• The method is consistent under weak assumptions on the classifier, provided the classifier can asymptotically distinguish the two distributions.
• To avoid data snooping, the authors recommend reporting results from multiple classifiers (e.g., logistic regression and random forests) and using out-of-sample accuracy to ensure robustness.

Experimental results

Research questions

• RQ1 Can a nonparametric test based on classification performance detect covariate imbalance more effectively than traditional methods that rely on marginal moments or mean differences?
• RQ2 How does the CPT compare in power to existing nonparametric tests like the Cross-Match and energy tests in detecting differences in multivariate distributions?
• RQ3 To what extent can 'black box' machine learning classifiers be used for valid, inference-driven hypothesis testing in social science applications?
• RQ4 Does the CPT provide a more reliable validation of 'as-if' random assignment in natural experiments, regression discontinuity designs, and matching methods compared to standard balance diagnostics?

Key findings

• The CPT consistently outperforms the Cross-Match and energy tests in detecting multivariate imbalance in Monte Carlo simulations, especially when imbalances are nonlinear or involve joint distributional differences.
• In real data applications, the CPT detected significant covariate imbalance in Eggers and Hainmueller (2009) that was missed by standard balance tests, casting new light on the validity of their regression discontinuity design.
• The CPT provided strong evidence supporting the 'judges design' identification strategy in Green and Winik (2010), confirming the robustness of their causal inference.
• The test maintains correct size (Type I error rate) and is consistent under weak assumptions, even when using flexible classifiers like random forests.
• Using out-of-sample accuracy instead of in-sample accuracy prevents degenerate behavior (e.g., perfect in-sample classification due to overfitting), ensuring valid inference.
• The method is robust to treatment imbalance when the test set is balanced across treatment and control groups, and the classifier is calibrated to a uniform prior on class labels.