[Paper Review] On Variational Bounds of Mutual Information

Ben Poole, Sherjil Ozair+3 more|arXiv (Cornell University)|May 16, 2019

Face and Expression Recognition153 citations

TL;DR

The paper unifies and extends variational bounds on mutual information (MI), introducing a continuum of lower bounds that trade bias and variance and showing their utility for estimation and representation learning in high dimensions.

ABSTRACT

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

Motivation & Objective

• Review and relate existing MI estimators within a single framework.
• Introduce a continuum of lower bounds that trade off bias and variance for MI estimation.
• Leverage conditional structure to derive tractable bounds that sandwich MI in representation learning contexts.
• Empirically characterize bias/variance of estimators and gradients on high-dimensional problems.
• Demonstrate the bounds in decoder-free disentangled representation learning on dSprites.

Proposed method

• Review and unify Barber & Agakov, Donsker–Varadhan, Nguyen–Wainwright–Jordan, MINE bounds within an unnormalized variational framework.
• Introduce the Tractable Unnormalized Barber–Agakov (TUBA) bound by upper bounding the log partition with a(y) to obtain a tractable lower bound on MI.
• Derive multi-sample bounds, recovering the NCE lower bound as a special case and connecting NWJ with multi-sample estimators.
• Propose a nonlinear interpolated bound I_alpha that blends m(y; x_{1:K}) and q(y) to trade off bias and variance (alpha in [0,1]).
• Present structured bounds when p(y|x) is tractable, including an InfoNCE bound with a known conditional and leave-one-out upper bounds.
• Show how to turn density-ratio estimators into MI bounds and discuss JS-based critics for stability.

Experimental results

Research questions

• RQ1 How do existing variational MI bounds relate within a single framework and what are their tradeoffs?
• RQ2 Can we construct a continuum of MI bounds that interpolate between low-bias/high-variance and high-bias/low-variance estimators?
• RQ3 How can known conditional structure p(y|x) be exploited to tighten MI bounds in representation learning?
• RQ4 What are the bias and variance properties of MI estimators and their gradients in high-dimensional settings?
• RQ5 Can MI bounds be effectively used for decoder-free disentangled representation learning?

Key findings

• A continuum of lower bounds (I_TUBA, I_NWJ, I_NCE, I_JS) allows trading bias and variance in MI estimation.
• The interpolated bound I_alpha bridges NWJ and NCE, enabling tunable bias-variance via alpha (0 to 1).
• Multi-sample bounds reproduce the NCE estimator as a special case and reduce variance through additional samples.
• Optimal critics can depend on the chosen bound; joint vs separable critics exhibit different variance characteristics in experiments.
• Upper and lower bounds can sandwich MI when p(y|x) is tractable, enabling practical bounds in representation learning.
• Decoder-free InfoMax_STYLE learning on dSprites demonstrates practical utility of the bounds for disentangled representations.