[Paper Review] Conditional independence testing: a predictive perspective

Marco Henrique de Almeida Inácio, Rafael Izbicki+1 more|arXiv (Cornell University)|Jul 31, 2019

Bayesian Modeling and Causal Inference12 references4 citations

TL;DR

This paper proposes COINP, a novel conditional independence test that evaluates feature relevance in supervised learning by comparing predictive performance between original and permuted data via a permutation-based p-value. It achieves superior statistical power and better type I error control—especially with small samples—by using rank-based inference instead of asymptotic approximations, outperforming existing methods like CPI in both simulated and real-world settings.

ABSTRACT

Conditional independence testing is a key problem required by many machine learning and statistics tools. In particular, it is one way of evaluating the usefulness of some features on a supervised prediction problem. We propose a novel conditional independence test in a predictive setting, and show that it achieves better power than competing approaches in several settings. Our approach consists in deriving a p-value using a permutation test where the predictive power using the unpermuted dataset is compared with the predictive power of using dataset where the feature(s) of interest are permuted. We conclude that the method achives sensible results on simulated and real datasets.

Motivation & Objective

• To develop a statistically valid and powerful method for testing conditional independence in supervised learning settings.
• To address limitations in existing methods, particularly poor type I error control and low power in small sample sizes.
• To provide a hypothesis test that answers 'is this feature relevant?' rather than quantifying feature importance.
• To improve upon prior approaches like CPI by using permutation-based p-values that are robust to outliers and distributional assumptions.
• To demonstrate the method's effectiveness on both simulated and real-world datasets, including the diamonds dataset.

Proposed method

• COINP tests the null hypothesis $ H_0: \mathbf{X}^S \perp Y \mid \mathbf{X}^{S^c} $ using a predictive framework that compares model performance on original versus permuted features.
• The method computes a p-value via a permutation test: the predictive risk of a model trained on original data is compared to the distribution of risks from models trained on permuted data.
• For each of $ B $ permutations, the features in $ \mathbf{X}^S $ are randomly shuffled across samples, preserving conditional independence under the null.
• The p-value is estimated as the proportion of permuted datasets where the predictive performance is at least as good as the original dataset.
• The approach is robust to outliers because it relies on ranks of predictive performance rather than raw loss values or t-tests.
• The method is implemented with various prediction models (e.g., linear regression, random forests, neural networks), with performance evaluated across multiple simulation settings.

Experimental results

Research questions

• RQ1 Does COINP maintain proper type I error control under the null hypothesis across diverse data distributions and sample sizes?
• RQ2 How does COINP's statistical power compare to CPI and other methods when testing conditional independence in high-dimensional or non-linear settings?
• RQ3 Can COINP achieve better power than existing methods, especially in small sample scenarios where asymptotic approximations fail?
• RQ4 Is COINP robust to outliers and model misspecification, particularly when using complex models like neural networks?
• RQ5 How does COINP's p-value distribution compare to uniformity under the null, and how does it compare to CPI in this regard?

Key findings

• COINP achieves proper p-value uniformity under the null hypothesis across a wide range of settings, including those with dependent covariates, where approximate methods fail.
• COINP shows significantly better control of type I error rates than CPI, especially in small sample sizes and with dependent covariates.
• COINP achieves higher statistical power than CPI across all simulation settings, with the largest gains observed when using linear regression models.
• The power of COINP increases with sample size, indicating consistency, as shown by higher power in $ n = 10,000 $ settings compared to $ n = 1,000 $.
• COINP is robust to outliers because it uses rank-based inference, unlike CPI which relies on t-tests on raw loss values and is sensitive to extreme values.
• In the diamonds dataset analysis, COINP correctly identifies all features as significant (p < 0.05) across models, while CPI shows inconsistent results, especially with neural networks.