[Paper Review] Intriguing Properties of Contrastive Losses

Ting Chen, Calvin Luo+1 more|arXiv (Cornell University)|Nov 5, 2020

Domain Adaptation and Few-Shot Learning37 references62 citations

TL;DR

The paper generalizes contrastive losses, studies instance-based learning with multiple objects, demonstrates local feature emergence, and reveals a feature-suppression phenomenon where easy-to-learn shared features can block learning of others.

ABSTRACT

We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.

Motivation & Objective

• Extend understanding of contrastive losses by proposing a generalized loss form with alignment and distribution terms.
• Assess whether instance-based (global) contrastive objectives can learn from images containing multiple objects and still yield meaningful local features.
• Investigate the feature suppression phenomenon where shared features across augmentations suppress learning of other features, and understand its implications for data augmentation.
• Construct controlled datasets to quantify competing features and analyze how easy-to-learn features influence representation learning.

Proposed method

• Propose a generalized contrastive loss of the form L_generalized = L_alignment + lambda L_distribution, showing standard NT-Xent as a special case.
• Use a variety of prior distributions for L_distribution, including uniform hypersphere, uniform hypercube, and normal priors, with distribution matching via LogSumExp or Sliced Wasserstein Distance (SWD).
• Relate alignment to reducing H(U|V) and distribution to maximizing entropy H(U) in a mutual-information framing.
• Train with deep, multi-layer projection heads to compare instantiations and assess sensitivity to batch size.
• Introduce SWD-based instantiations to enable diverse priors beyond uniform hypersphere.
• Experiment with CIFAR-10 and ImageNet under SimCLR-style setups to compare loss variants and projection head depths.

Experimental results

Research questions

• RQ1 Can generalized contrastive losses with various priors perform similarly when a deep projection head is used?
• RQ2 Do instance-based (global) contrastive objectives learn meaningful local features when images contain multiple objects?
• RQ3 How does the presence of competing features shared across augmented views affect contrastive learning, and can easy-to-learn features suppress other features?
• RQ4 Can controlled datasets reveal the extent of feature suppression and the limitations of current augmentations?

Key findings

• Differences between generalized contrastive losses are small with a deep projection head across CIFAR-10 and ImageNet.
• Instance-based learning can learn from images with multiple objects and recover meaningful local features despite operating on global representations.
• Baselining with datasets containing explicit competing features shows that a few bits of easy-to-learn shared information can suppress learning of other features, sometimes fully blocking them.
• Dominant objects can suppress learning of smaller objects in multi-object scenarios, revealing an open challenge for contrastive methods under real-world clutter.
• Extra easy-to-learn information added to views (e.g., random bits) can completely disable contrastive learning, whereas generative models like VAEs are less affected.
• The study emphasizes that data augmentation design critically influences what features are learned and highlights feature suppression as a fundamental limitation of current contrastive approaches.