[Paper Review] Formal Limitations on the Measurement of Mutual Information

David McAllester, Karl Stratos|arXiv (Cornell University)|Nov 10, 2018

Machine Learning and Algorithms27 references66 citations

TL;DR

The paper proves universal statistical limitations on distribution-free high-confidence lower bounds for mutual information, KL divergence, and entropy when estimated from finite samples, and proposes a practical difference-of-entropies (DoE) estimator that can meaningfully estimate large MI in practice.

ABSTRACT

Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).

Motivation & Objective

• Motivate the difficulty of estimating mutual information from finite data and the limitations of lower-bound estimators.
• Provide formal results showing distribution-free high-confidence lower bounds cannot exceed logarithmic growth in sample size.
• Unify and generalize prior negative results on KL divergence and entropy estimation.
• Propose the difference-of-entropies (DoE) approach as a practical alternative for estimating mutual information.
• Demonstrate empirically that DoE can yield meaningful MI estimates on synthetic and real data.

Proposed method

• Analyze lower bounds on KL divergence via the Donsker-Varadhan framework and show they cannot exceed O(log N) in sample size N.
• Prove a distribution-free lower-bound limitation for entropy estimation that also scales as O(log N).
• Demonstrate that these limitations imply universal constraints on any distribution-free MI estimators.
• Propose expressing MI as a difference of entropies and estimate the entropy terms via cross-entropy minimization.
• Introduce the DoE estimator, which computes I(X;Y) as H(pX, qX) − H(pX|Y, qX|Y) by minimizing cross-entropy losses.
• Provide theoretical discussion on why cross-entropy upper bounds can be informative for large entropies, and empirical evidence of DoE’s performance.

Experimental results

Research questions

• RQ1 Can a distribution-free high-confidence lower bound on mutual information, estimated from N samples, exceed a logarithmic function of N?
• RQ2 Do lower bounds on KL divergence or entropy necessarily bound MI in a way that prevents accurate estimation from finite samples?
• RQ3 Is there a practical estimator for MI that avoids the logarithmic barrier and remains useful in real-world large-MI settings?
• RQ4 Does expressing MI as a difference of entropies and estimating via cross-entropy provide a more robust alternative than variational lower bounds?
• RQ5 How do DoE-based estimates compare to existing estimators on synthetic and real datasets?

Key findings

• Any distribution-free high-confidence lower bound on MI estimated from N samples cannot exceed a function on the order of ln N.
• Lower bounds on KL divergence and entropy suffer the same logarithmic limitation, which constrains universal MI estimation.
• The DoE estimator, based on cross-entropy bounds, can meaningfully estimate large MI in practice, without formal lower-bound guarantees.
• DoE outperforms existing variational lower-bound estimators on synthetic correlated data and real-world data in their experiments.
• Empirical results show large MI (e.g., over 120 bits for related article pairs and 54 bits for translation pairs) when using DoE, with shuffled pairs showing near-zero MI.