[Paper Review] On Mutual Information in Contrastive Learning for Visual Representations

Mike Wu, Chengxu Zhuang+3 more|arXiv (Cornell University)|May 27, 2020

Advanced Image and Video Retrieval Techniques19 references49 citations

TL;DR

The paper presents an information-theoretic view of contrastive learning, showing that IR, LA, CMC, and related methods are lower bounds on mutual information between image views; it introduces flexible negative-sample strategies (VINCE, BALL, RING, CAVE) and shows they improve transfer performance across vision tasks.

ABSTRACT

In recent years, several unsupervised, "contrastive" learning algorithms in vision have been shown to learn representations that perform remarkably well on transfer tasks. We show that this family of algorithms maximizes a lower bound on the mutual information between two or more "views" of an image where typical views come from a composition of image augmentations. Our bound generalizes the InfoNCE objective to support negative sampling from a restricted region of "difficult" contrasts. We find that the choice of negative samples and views are critical to the success of these algorithms. Reformulating previous learning objectives in terms of mutual information also simplifies and stabilizes them. In practice, our new objectives yield representations that outperform those learned with previous approaches for transfer to classification, bounding box detection, instance segmentation, and keypoint detection. % experiments show that choosing more difficult negative samples results in a stronger representation, outperforming those learned with IR, LA, and CMC in classification, bounding box detection, instance segmentation, and keypoint detection. The mutual information framework provides a unifying comparison of approaches to contrastive learning and uncovers the choices that impact representation learning.

Motivation & Objective

• Provide a unified information-theoretic view of contrastive visual representation learning through mutual information (MI) bounds.
• Show how data augmentations (views) and negative sample choices critically affect representation quality.
• Derive and analyze lower bounds (InfoNCE and VINCE) and connect them to existing methods (IR, LA, CMC, SimCLR).
• Propose and evaluate new negative-sample strategies (BALL, RING, CAVE) and annealing to improve transfer performance.

Proposed method

• Formulate contrastive objectives as MI estimators, including a generalization of InfoNCE to allow negative sampling from restricted distributions (VINCE).
• Show equivalence of IR and CMC to InfoNCE under extended view and memory-bank formulations, and simplify IR to a memory-free (IR-nce) variant for stability.
• Introduce and analyze BALL, RING, and CAVE negative-sample schemes that sample negatives from constrained regions (balls/rings) around current views.
• Propose annealing of the negative-distribution parameters to balance learning signal and stability.
• Relate Local Aggregation (LA) to VINCE, and derive conditions under which LA bounds MI between weighted view sets.
• Provide practical variants (LA-nce, BALL Lab, etc.) and show how these relate to stability and performance.

Experimental results

Research questions

• RQ1 How do different view (augmentation) choices affect the quality of learned representations in contrastive learning?
• RQ2 How does the distribution of negative samples impact the mutual information bound and downstream transfer performance?
• RQ3 Can we unify IR, LA, and CMC under mutual-information-based objectives and what practical variants emerge from this view?
• RQ4 Do constrained (ball/ring/cave) negative-sample strategies improve transfer tasks like image classification and detection compared to standard InfoNCE?
• RQ5 What is the role of annealing in negative-sample distributions for stabilizing and improving learning?

Key findings

• Mutual-information view shows IR, LA, CMC, and SimCLR are lower bounds on MI between image views, with InfoNCE as a primary estimator.
• Harder, more informative negative samples (BALL, RING, CAVE) improve transfer performance over IR-nce across CIFAR10 and ImageNet- and COCO-based tasks.
• Simplifying IR to InfoNCE improves stability and sometimes performance; memory-bank can be removed in some settings without loss.
• Introducing variational negative sampling (VINCE) enables new ties between LA and MI, and shows LA is bounded by BALL/VINCE.
• Annealing negative-sample distributions further improves results, yielding state-competitive performance across multiple transfer tasks.
• Best reported variants (e.g., RING Lab+s-neigh+anneal) outperform IR, LA, and CMC on several tasks.