[Paper Review] Theoretical Foundations of Equitability and the Maximal Information Coefficient

Yakir Reshef, David N. Reshef+2 more|arXiv (Cornell University)|Aug 21, 2014

Bayesian Modeling and Causal Inference27 references10 citations

TL;DR

This paper formalizes the theoretical foundations of equitability and the maximal information coefficient (MIC), introducing a population-level version of MIC called MIC* to clarify its relationship with mutual information. It establishes MIC* as a continuous, canonical smoothing of mutual information and provides a rigorous framework that generalizes equitability as a property of dependence measures, resolving ambiguities in prior work and enabling new estimators and computational methods.

ABSTRACT

The maximal information coefficient (MIC) is a tool for finding the strongest pairwise relationships in a data set with many variables (Reshef et al., 2011). MIC is useful because it gives similar scores to equally noisy relationships of different types. This property, called {\em equitability}, is important for analyzing high-dimensional data sets. Here we formalize the theory behind both equitability and MIC in the language of estimation theory. This formalization has a number of advantages. First, it allows us to show that equitability is a generalization of power against statistical independence. Second, it allows us to compute and discuss the population value of MIC, which we call MIC_*. In doing so we generalize and strengthen the mathematical results proven in Reshef et al. (2011) and clarify the relationship between MIC and mutual information. Introducing MIC_* also enables us to reason about the properties of MIC more abstractly: for instance, we show that MIC_* is continuous and that there is a sense in which it is a canonical "smoothing" of mutual information. We also prove an alternate, equivalent characterization of MIC_* that we use to state new estimators of it as well as an algorithm for explicitly computing it when the joint probability density function of a pair of random variables is known. Our hope is that this paper provides a richer theoretical foundation for MIC and equitability going forward. This paper will be accompanied by a forthcoming companion paper that performs extensive empirical analysis and comparison to other methods and discusses the practical aspects of both equitability and the use of MIC and its related statistics.

Motivation & Objective

• To provide a formal theoretical framework for equitability, a concept that aims to assign similar scores to relationships of equal noise level regardless of type.
• To define and analyze the population version of MIC, denoted MIC*, as a canonical smoothing of mutual information.
• To clarify the mathematical relationship between MIC and mutual information, showing that MIC* generalizes and strengthens prior results.
• To resolve ambiguities in the original definition of equitability by introducing a precise, generalizable formulation grounded in estimation theory.
• To enable new computational and statistical tools by deriving an alternate, equivalent characterization of MIC* that supports explicit computation and estimation.

Proposed method

• Introduces MIC* as the population limit of MIC, defined as the supremum of normalized mutual information over all finite grids partitioning the joint distribution of a pair of random variables.
• Uses estimation theory to formalize equitability as a generalization of statistical power against independence, linking it to effect size estimation.
• Applies information-theoretic tools, including entropy and binary entropy functions, to bound deviations in mutual information under perturbations of the joint distribution.
• Derives a new characterization of MIC* using grid-based approximations and proves that MIC* is continuous with respect to the underlying distribution.
• Employs lemmas bounding entropy differences under probability mass perturbations (e.g., Lemma A.7) to establish stability and convergence properties of MIC*.
• Proposes new estimators of MIC* based on the derived characterization, enabling practical computation when the joint density is known.

Experimental results

Research questions

• RQ1 What is the precise theoretical definition of equitability, and how can it be formalized in a way that generalizes across different types of relationships?
• RQ2 How does MIC* relate to mutual information, and in what sense is it a canonical smoothing of mutual information?
• RQ3 Can MIC* be computed explicitly when the joint probability density function is known, and what are the computational and statistical properties of such a computation?
• RQ4 What are the theoretical guarantees for MIC* in terms of continuity and stability under small perturbations of the underlying distribution?
• RQ5 How does the formalization of equitability via MIC* resolve ambiguities and limitations in the original definition of equitability presented in prior work?

Key findings

• MIC* is formally defined as the supremum of normalized mutual information over all finite grids, providing a population-level benchmark for MIC.
• MIC* is continuous with respect to the underlying joint distribution, ensuring stable behavior under small changes in data-generating mechanisms.
• MIC* generalizes and strengthens the mathematical results from the original MIC paper by providing a rigorous link to mutual information.
• An alternate, equivalent characterization of MIC* is derived, enabling explicit computation when the joint density is known and supporting new estimation algorithms.
• The formal framework shows that equitability is a generalization of statistical power against independence, embedding it within a broader estimation-theoretic context.
• Lemmas on entropy perturbations (e.g., Lemma A.7) establish that MIC* is robust to small changes in the joint distribution, with bounds involving binary entropy and log-k terms.