[Paper Review] On Mutual Information Maximization for Representation Learning

Michael Tschannen, Josip Djolonga+3 more|arXiv (Cornell University)|Jul 31, 2019

Domain Adaptation and Few-Shot Learning49 references219 citations

TL;DR

The paper questions MI maximization as a sole objective for unsupervised representation learning, showing estimator and architecture biases largely drive learned representations and linking the ideas to deep metric learning.

ABSTRACT

Many recent methods for unsupervised or self-supervised representation learning train feature extractors by maximizing an estimate of the mutual information (MI) between different views of the data. This comes with several immediate problems: For example, MI is notoriously hard to estimate, and using it as an objective for representation learning may lead to highly entangled representations due to its invariance under arbitrary invertible transformations. Nevertheless, these methods have been repeatedly shown to excel in practice. In this paper we argue, and provide empirical evidence, that the success of these methods cannot be attributed to the properties of MI alone, and that they strongly depend on the inductive bias in both the choice of feature extractor architectures and the parametrization of the employed MI estimators. Finally, we establish a connection to deep metric learning and argue that this interpretation may be a plausible explanation for the success of the recently introduced methods.

Motivation & Objective

• Motivate unsupervised representation learning via information-theoretic goals and assess the role of mutual information (MI).
• Show that maximizing MI bounds can bias encoders toward undesired representations.
• Demonstrate that estimator choice and encoder architecture strongly influence downstream performance.
• Provide an alternative interpretation by connecting MI-based methods to deep metric learning and triplet losses.

Proposed method

• Formulate representation learning as maximizing an MI lower bound between two views of data using estimators like InfoNCE and NWJ.
• Experiment with invertible and non-invertible encoders to observe how MI maximization affects downstream tasks.
• Vary critic architectures (bilinear, separable, MLP) to study their impact on learned representations.
• Compare encoder architectures (MLP vs ConvNet) while matching MI bounds to isolate architectural effects.
• Analyze the role of negative sampling in InfoNCE and NWJ and its effect on MI estimation and performance.
• Relate MI-based objectives to triplet-based metric learning losses to reinterpret results.

Experimental results

Research questions

• RQ1 Does maximizing MI (via common estimators) reliably yield useful representations for downstream tasks?
• RQ2 How do encoder architecture and estimator choice bias the learned representations?
• RQ3 What is the role of the critic architecture and negative sampling in MI-based representation learning?
• RQ4 Can the observed success of MI-based methods be better explained by deep metric learning principles?
• RQ5 Under what conditions do looser MI bounds yield better representations?

Key findings

• MI maximization does not guarantee good representations; some invertible encoders maximizing MI yield worse downstream performance than raw pixels.
• Estimators like InfoNCE and NWJ bias encoders toward hard-to-invert or ill-conditioned mappings, affecting representations.
• Higher-capacity critics can tighten MI bounds yet harm downstream performance, while simpler critics (bilinear/separable) can improve it.
• Encoder architecture often has a larger impact than the specific MI estimator when achieving the same MI bound.
• A metric-learning perspective via triplet losses provides an alternative explanation for empirical success, questioning the primacy of MI as the objective.