[Paper Review] MINE: Mutual Information Neural Estimation

Mohamed Ishmael Belghazi, Aristide Baratin+5 more|arXiv (Cornell University)|Jan 12, 2018

Adversarial Robustness in Machine Learning305 citations

TL;DR

Introduces a neural estimator for mutual information (MINE) using dual KL-divergence representations, scalable to high dimensions and trainable by back-propagation, with applications to GANs, ALI, and information bottleneck.

ABSTRACT

We argue that the estimation of mutual information between high dimensional continuous random variables can be achieved by gradient descent over neural networks. We present a Mutual Information Neural Estimator (MINE) that is linearly scalable in dimensionality as well as in sample size, trainable through back-prop, and strongly consistent. We present a handful of applications on which MINE can be used to minimize or maximize mutual information. We apply MINE to improve adversarially trained generative models. We also use MINE to implement Information Bottleneck, applying it to supervised classification; our results demonstrate substantial improvement in flexibility and performance in these settings.

Motivation & Objective

• Motivate and address the challenge of estimating mutual information for high-dimensional continuous variables.
• Propose a neural estimator (MINE) based on the Donsker-Varadhan and f-divergence dual representations.
• Demonstrate strong consistency and favorable sample complexity of MINE.
• Apply MINE to mitigate mode collapse in GANs, enhance reconstruction in ALI, and enable continuous information bottleneck methods.

Proposed method

• Define a neural network T_theta over X and Z that serves as the dual function for KL-divergence.
• Use the bound I(X;Z) ≥ E_{P_XZ}[T_theta] − log E_{P_X ⊗ P_Z}[e^{T_theta}].
• Estimate I(X;Z) by optimizing over theta with minibatch samples from P_{XZ} and from P_X ⊗ P_Z.
• Address SGD gradient bias in the denominator by using a moving-average bias correction.
• Propose MINE (and MINE-f) as estimators; discuss bias correction and practical training details.
• Provide theoretical results: strong consistency (approximation and estimation lemmas) and a sample-complexity bound.

Experimental results

Research questions

• RQ1 Can a neural network-based dual representation estimate mutual information between high-dimensional variables accurately?
• RQ2 Is MINE strongly consistent and what is its sample complexity?
• RQ3 Can MINE be effectively used to maximize or minimize mutual information in practical learning settings (GANs, ALI, IB)?

Key findings

• MINE is strongly consistent: it can approximate I(X;Z) arbitrarily well with sufficient network capacity and samples.
• Empirical results show MINE closely matches ground-truth MI for low-dimensions and outperforms nonparametric k-NN in higher dimensions.
• Maximizing mutual information with MINE improves mode coverage in GANs and ALI reconstructions, and helps reduce mode collapse.
• MINE enables continuous information bottleneck implementations that outperform variational bottleneck baselines in several setups.
• On Stacked MNIST, MINE achieves full mode coverage with competitive KL divergence to the data distribution.
• MINE-based IB experiments on permutation-invariant MNIST demonstrate favorable misclassification rates compared to baselines.