[Paper Review] Efficient Estimation of Mutual Information for Strongly Dependent Variables

Shuyang Gao, Greg Ver Steeg+1 more|arXiv (Cornell University)|Nov 7, 2014

Complex Network Analysis Techniques31 references89 citations

TL;DR

This paper proposes a novel k-nearest-neighbor (kNN)-based mutual information (MI) estimator that corrects for local non-uniformity in the joint distribution, a key flaw in existing kNN estimators. Unlike previous methods that require exponentially many samples for strong dependencies, the new estimator achieves accurate MI estimation with significantly fewer samples, especially in high-dependency regimes.

ABSTRACT

We demonstrate that a popular class of nonparametric mutual information (MI) estimators based on k-nearest-neighbor graphs requires number of samples that scales exponentially with the true MI. Consequently, accurate estimation of MI between two strongly dependent variables is possible only for prohibitively large sample size. This important yet overlooked shortcoming of the existing estimators is due to their implicit reliance on local uniformity of the underlying joint distribution. We introduce a new estimator that is robust to local non-uniformity, works well with limited data, and is able to capture relationship strengths over many orders of magnitude. We demonstrate the superior performance of the proposed estimator on both synthetic and real-world data.

Motivation & Objective

• To identify the critical flaw in existing kNN-based MI estimators that causes poor performance under strong dependence.
• To address the exponential sample size requirement of current estimators when mutual information is high.
• To develop a new non-parametric MI estimator that remains accurate even with limited data and strong relationships.
• To provide a practical, open-source tool for efficient MI estimation in real-world data mining applications.

Proposed method

• Introduces a correction term to account for local non-uniformity in the joint density, which previous kNN estimators implicitly assume away.
• Modifies the standard kNN entropy estimator by incorporating a bias correction that adjusts for density estimation errors near boundaries and in non-uniform regions.
• Derives a new mutual information estimator based on the corrected kNN entropy estimates, using the standard identity I(X) = ΣH(Xi) - H(X).
• Employs a modified kNN graph structure that better captures local density variations, especially in high-dependency scenarios.
• Uses a weighted combination of nearest-neighbor distances to estimate local densities while accounting for non-uniformity through a correction factor.
• Validates the method using synthetic data with known MI values and real-world datasets to demonstrate robustness and sample efficiency.

Experimental results

Research questions

• RQ1 Why do existing kNN-based MI estimators fail to accurately estimate mutual information when the true MI is high?
• RQ2 What underlying assumption in current kNN estimators leads to poor performance under strong dependence?
• RQ3 Can a correction term for local non-uniformity improve the accuracy of kNN-based MI estimation with limited samples?
• RQ4 How does the proposed estimator compare to state-of-the-art methods in terms of sample efficiency and accuracy across varying levels of dependence?

Key findings

• The proposed estimator achieves accurate MI estimation with orders-of-magnitude fewer samples than required by standard kNN estimators in high-dependency regimes.
• Existing kNN-based MI estimators require exponentially many samples as mutual information increases, making strong relationships hard to measure.
• The new estimator significantly outperforms the KSG estimator and other kNN variants in estimating strong nonlinear relationships with small sample sizes.
• The method is robust to local non-uniformity and boundary effects, which commonly bias traditional kNN estimators.
• Empirical results on synthetic and real-world data show the proposed estimator maintains high accuracy even when true MI exceeds 5 nats.
• The open-source implementation demonstrates practical utility for data mining tasks involving high-dimensional, strongly dependent variables.