[Paper Review] Understanding the Limitations of Variational Mutual Information Estimators

Jiaming Song, Stefano Ermon|arXiv (Cornell University)|Oct 14, 2019

Neural Networks and ApplicationsComputer Science30 references87 citations

TL;DR

The paper analyzes variance and bias issues in variational mutual information estimators (e.g., MINE, NWJ, CPC), reveals self-consistency failures, and proposes a variance-reduced estimator SMILE, along with a unified optimization view over density ratios.

ABSTRACT

Variational approaches based on neural networks are showing promise for estimating mutual information (MI) between high dimensional variables. However, they can be difficult to use in practice due to poorly understood bias/variance tradeoffs. We theoretically show that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. We also empirically demonstrate that existing estimators fail to satisfy basic self-consistency properties of MI, such as data processing and additivity under independence. Based on a unified perspective of variational approaches, we develop a new estimator that focuses on variance reduction. Empirical results on standard benchmark tasks demonstrate that our proposed estimator exhibits improved bias-variance trade-offs on standard benchmark tasks.

Motivation & Objective

• Motivate reliable mutual information (MI) estimation in high-dimensional settings for tasks like representation learning and RL.
• Analyze bias/variance tradeoffs of variational MI estimators (MINE, NWJ, CPC) and identify fundamental limitations.
• Propose a unified perspective that frames MI estimation as density ratio optimization and highlight partition-function estimation as a source of high variance.
• Introduce a variance-reduction strategy (SMILE) for MI estimation.
• Empirically evaluate estimators on benchmarks and self-consistency tests to assess practical reliability.

Proposed method

• Formulate MI estimation as a constrained optimization over valid density ratios with respect to Q (the product of marginals).
• Show that the KL divergence D_KL(P||Q) equals the supremum of E_P[log r] over r in the density-ratio family Delta(Q).
• Review discriminative (MINE, NWJ, CPC) and generative (BA, GM) MI estimators and explain how they fit into the density-ratio optimization framework.
• Demonstrate that NWJ/MINE can have variance that grows exponentially with true MI due to partition-function estimation.
• Propose SMILE: a clipped (log-density) ratio estimator leading to reduced variance, with I_SMILE defined as E_P[T] - log E_Q[clip(e^T, e^{-tau}, e^{tau})].
• Discuss bias-variance trade-offs of SMILE and analyze theoretical properties (bias bounds and variance bounds) under clipping.
• Provide self-consistency tests (independence, data processing, additivity) to evaluate MI estimators on high-dimensional data.

Experimental results

Research questions

• RQ1 How do variational MI estimators behave in terms of bias and variance as the true MI increases?
• RQ2 Do common MI estimators satisfy fundamental self-consistency properties of MI (data processing, additivity, independence) in practice?
• RQ3 Can a unified view of MI estimation as density-ratio optimization explain variance issues and guide improvements?
• RQ4 Does a variance-reduction technique (SMILE) improve the bias-variance trade-off relative to existing estimators?
• RQ5 How do different estimators perform on standard benchmarks and self-consistency tests using high-dimensional data (images)?

Key findings

• Estimators like NWJ and MINE can exhibit variance that grows exponentially with the ground-truth MI, leading to poor bias-variance trade-offs.
• Many variational MI estimators fail basic MI self-consistency tests such as data processing and additivity under independence when evaluated on image data.
• A unified perspective shows MI estimation as optimization over valid density ratios; high variance is tied to partition-function estimation.
• The proposed SMILE estimator reduces variance via clipping of density ratios, yielding improved bias-variance trade-offs on benchmark tasks (tau controls bias-variance).
• CPC shows lower variance but higher bias; SMILE can achieve substantially lower variance with comparable bias to NWJ, depending on tau.
• Generative approaches perform poorly at low MI, while discriminative approaches perform poorly at high MI, indicating limitations in using these estimators as proxies for MI optimization.