[Paper Review] On Distance and Kernel Measures of Conditional Independence

Tianhong Sheng, Bharath K. Sriperumbudur|arXiv (Cornell University)|Dec 2, 2019

Bayesian Modeling and Causal Inference28 references4 citations

TL;DR

This paper establishes a theoretical connection between distance-based and kernel-based measures of conditional independence, showing that generalized conditional distance covariance (gCdCov) is equivalent to a kernel-based measure (HSCIC) under certain conditions. However, popular kernel-based conditional independence measures like HSCIC do not generally have a simple distance representation, except in limiting cases, revealing a key distinction from joint independence where equivalence holds.

ABSTRACT

Measuring conditional independence is one of the important tasks in statistical inference and is fundamental in causal discovery, feature selection, dimensionality reduction, Bayesian network learning, and others. In this work, we explore the connection between conditional independence measures induced by distances on a metric space and reproducing kernels associated with a reproducing kernel Hilbert space (RKHS). For certain distance and kernel pairs, we show the distance-based conditional independence measures to be equivalent to that of kernel-based measures. On the other hand, we also show that some popular---in machine learning---kernel conditional independence measures based on the Hilbert-Schmidt norm of a certain cross-conditional covariance operator, do not have a simple distance representation, except in some limiting cases. This paper, therefore, shows the distance and kernel measures of conditional independence to be not quite equivalent unlike in the case of joint independence as shown by Sejdinovic et al. (2013).

Motivation & Objective

• To investigate the theoretical relationship between distance-based and kernel-based measures of conditional independence.
• To generalize conditional distance covariance (CdCov) to arbitrary metric spaces of negative type, introducing generalized CdCov (gCdCov).
• To establish equivalence between gCdCov and a kernel-based conditional independence measure (HSCIC).
• To examine whether popular kernel-based measures (e.g., HSCIC) can be represented via distance metrics.
• To compare the statistical and computational properties of empirical estimators for both classes of measures.

Proposed method

• Generalize CdCov to metric spaces of negative type, defining generalized conditional distance covariance (gCdCov).
• Define a kernel-based conditional independence measure (HSCIC) using the Hilbert-Schmidt norm of the conditional cross-covariance operator.
• Establish equivalence between gCdCov and HSCIC under specific kernel and distance pairs, using reproducing kernel Hilbert space (RKHS) embeddings.
• Derive empirical estimators for both gCdCov and HSCIC using kernel matrices and hat matrices.
• Use concentration of measure in Hilbert spaces to establish consistency of the empirical HSCIC estimator.
• Demonstrate that HSCIC does not require density estimation, unlike distance-based estimators, while maintaining similar computational complexity.

Experimental results

Research questions

• RQ1 Under what conditions is generalized conditional distance covariance (gCdCov) equivalent to a kernel-based conditional independence measure (HSCIC)?
• RQ2 Can popular kernel-based conditional independence measures such as HSCIC be represented as distance-based measures in general, or only in limiting cases?
• RQ3 How do the empirical estimators of gCdCov and HSCIC compare in terms of computational complexity and statistical consistency?
• RQ4 Is there a fundamental difference between conditional and joint independence in terms of the equivalence of distance and kernel measures?
• RQ5 Can the kernel-based HSCIC be viewed as a generalization of distance-based CdCov?

Key findings

• Generalized conditional distance covariance (gCdCov) is equivalent to the kernel-based HSCIC measure under specific conditions involving kernels of negative type and metrics of negative type.
• The HSCIC estimator is consistent and does not require density estimation, unlike the empirical gCdCov estimator.
• Empirical HSCIC and gCdCov estimators have similar computational complexity, despite different underlying assumptions.
• Popular kernel-based conditional independence measures such as HSCIC do not generally admit a simple distance representation, except in limiting cases.
• The equivalence between distance and kernel measures of conditional independence does not hold universally, unlike in the case of joint independence as shown by Sejdinovic et al. (2013).
• The HSCIC estimator can be expressed using kernel matrices and hat matrices, with a closed-form trace-based expression involving KZ, HY, and HX.