[Paper Review] Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information

Jakob Runge|arXiv (Cornell University)|Sep 5, 2017

Bayesian Modeling and Causal Inference66 citations

TL;DR

A fully non-parametric test for conditional independence of continuous variables using a nearest-neighbor estimator of conditional mutual information (CMI) combined with a local permutation scheme, enabling adaptation to highly localized densities and nonlinear dependencies.

ABSTRACT

Conditional independence testing is a fundamental problem underlying causal discovery and a particularly challenging task in the presence of nonlinear and high-dimensional dependencies. Here a fully non-parametric test for continuous data based on conditional mutual information combined with a local permutation scheme is presented. Through a nearest neighbor approach, the test efficiently adapts also to non-smooth distributions due to strongly nonlinear dependencies. Numerical experiments demonstrate that the test reliably simulates the null distribution even for small sample sizes and with high-dimensional conditioning sets. The test is better calibrated than kernel-based tests utilizing an analytical approximation of the null distribution, especially for non-smooth densities, and reaches the same or higher power levels. Combining the local permutation scheme with the kernel tests leads to better calibration, but suffers in power. For smaller sample sizes and lower dimensions, the test is faster than random fourier feature-based kernel tests if the permutation scheme is (embarrassingly) parallelized, but the runtime increases more sharply with sample size and dimensionality. Thus, more theoretical research to analytically approximate the null distribution and speed up the estimation for larger sample sizes is desirable.

Motivation & Objective

• Address conditional independence testing for continuous, possibly nonlinear and high-dimensional data.
• Develop a fully non-parametric CI test based on conditional mutual information (CMI).
• Introduce a local, nearest-neighbor permutation scheme to generate the null distribution without destroying Z-dependence.
• Demonstrate better calibration and competitive power compared with kernel-based CI tests across various scenarios.

Proposed method

• Estimate CMI using a Kozachenko-Leonenko-type k-nearest neighbor entropy estimator in the joint X,Y,Z space with k specified as k_CMI.
• Compute the CMI estimator: Ĩ_{XY|Z} = ψ(k) + (1/n) sum_i [ψ(k_{Z,i}) - ψ(k_{XZ,i}) - ψ(k_{YZ,i})].
• For each sample, determine ε_i from the k_CMI-th nearest neighbor in the joint space and count k_{XZ,i}, k_{YZ,i}, k_{Z,i} in respective subspaces using ε_i.
• Employ a local nearest-neighbor permutation scheme to generate surrogates: map each X-value to a random neighbor in Z while preserving Z and X–Z structure.
• Generate the null distribution via B surrogates and compute p-values as the proportion of surrogates with Ĩ ≥ Ĩ(X;Y|Z).
• Offer KD-tree-based nearest-neighbor search for efficiency; note the method’s data-adaptive behavior and linear-ish scaling in k and D.

Experimental results

Research questions

• RQ1 Can a fully non-parametric CI test for continuous data be constructed using a nearest-neighbor estimator of CMI that remains reliable with small sample sizes and high-dimensional Z?
• RQ2 Does a local permutation scheme based on nearest neighbors provide well-calibrated null distributions and competitive power relative to kernel-based CI tests under nonlinear dependencies?
• RQ3 How do choices of k_CMI and k_perm affect calibration, power, and runtime across varying sample sizes and conditioning set dimensions?
• RQ4 Is the proposed CMIknn method robust to highly localized densities and non-smooth densities arising from nonlinear dependencies?

Key findings

• CMIknn yields well-calibrated null distributions for small to moderate sample sizes and high-dimensional conditioning, outperforming several kernel-based tests in calibration under challenging densities.
• The local permutation scheme improves calibration compared with full or analytical null approximations, particularly for small n.
• CMIknn achieves comparable or higher power than KCIT, RCIT, RCoT, and CDC in many nonlinear settings, with especially strong performance for highly nonlinear dependencies.
• For small sample sizes, CMIknn can be faster than RCIT/RCoT when using KD-tree neighbor searches, though the permutation step adds substantial computational load.
• Calibration can be further improved by combining kernel tests with the local permutation scheme, though at a cost to power.
• The authors provide freely available code implementing CMIknn at a GitHub repository.