[Paper Review] Estimating Mutual Information for Discrete-Continuous Mixtures

Weihao Gao, Sreeram Kannan+2 more|arXiv (Cornell University)|Sep 19, 2017

Bayesian Modeling and Causal InferenceComputer Science5 references77 citations

TL;DR

The paper introduces a mutual information estimator for mixed discrete-continuous distributions based on Radon-Nikodym derivatives and k-nearest neighbors, proves consistency, and shows favorable performance over baselines.

ABSTRACT

Estimating mutual information from observed samples is a basic primitive, useful in several machine learning tasks including correlation mining, information bottleneck clustering, learning a Chow-Liu tree, and conditional independence testing in (causal) graphical models. While mutual information is a well-defined quantity in general probability spaces, existing estimators can only handle two special cases of purely discrete or purely continuous pairs of random variables. The main challenge is that these methods first estimate the (differential) entropies of X, Y and the pair (X;Y) and add them up with appropriate signs to get an estimate of the mutual information. These 3H-estimators cannot be applied in general mixture spaces, where entropy is not well-defined. In this paper, we design a novel estimator for mutual information of discrete-continuous mixtures. We prove that the proposed estimator is consistent. We provide numerical experiments suggesting superiority of the proposed estimator compared to other heuristics of adding small continuous noise to all the samples and applying standard estimators tailored for purely continuous variables, and quantizing the samples and applying standard estimators tailored for purely discrete variables. This significantly widens the applicability of mutual information estimation in real-world applications, where some variables are discrete, some continuous, and others are a mixture between continuous and discrete components.

Motivation & Objective

• Motivate accurate mutual information estimation when variables are mixed discrete and continuous.
• Develop a direct estimator that works in general measure spaces where entropy is not well-defined.
• Establish theoretical guarantees (consistency) for the estimator.
• Demonstrate practical performance against standard baselines on synthetic and real data.

Proposed method

• Define MI for general distributions via the Radon-Nikodym derivative.
• Propose a mixed-variable MI estimator that uses k-NN distances to estimate the derivative at each sample.
• Handle discrete points, joint-density regions, and purely continuous parts within a unified scheme.
• Prove estimator consistency under mild technical conditions (ℓ2-consistency).
• Show that the estimator recovers special cases (purely discrete, purely continuous, or mixed) as limits of the method.
• Compare empirically against discretization, adaptive partitioning, and noisy-KSG baselines.

Experimental results

Research questions

• RQ1 Can MI be consistently estimated when X and/or Y are mixtures of discrete and continuous components?
• RQ2 How does a Radon-Nikodym-based estimator perform relative to 3H-based and KSG-style methods in mixed regimes?
• RQ3 Does the proposed estimator adapt to purely discrete, purely continuous, and mixed settings?
• RQ4 What is the empirical performance of the estimator on synthetic and real-world mixed data tasks?

Key findings

• The estimator is consistent in ℓ2 under technical assumptions and common practical distributions.
• It outperforms baselines that discretize or add noise to mix variables across a range of experiments.
• The method naturally recovers purely discrete and purely continuous cases as special instances.
• Experiments include higher-dimensional mixtures and zero-inflated distributions, showing robust performance.
• Applications include feature selection and gene regulatory network inference under dropout-like corruption.
• Empirical results demonstrate favorable sample-efficiency compared with discretization and noisy-KSG approaches.