[Paper Review] Conditional Negative Sampling for Contrastive Learning of Visual Representations

Mike Wu, Milan Mosse+3 more|arXiv (Cornell University)|May 3, 2021

Domain Adaptation and Few-Shot Learning46 references12 citations

TL;DR

This paper proposes conditional negative sampling for contrastive learning, where negatives are selected semi-hardly around each positive sample to improve mutual information estimation. By conditioning negative sampling on the positive, the method reduces variance and boosts performance, achieving 2–5% absolute accuracy gains across multiple models and benchmarks under linear evaluation and transfer learning.

ABSTRACT

Recent methods for learning unsupervised visual representations, dubbed contrastive learning, optimize the noise-contrastive estimation (NCE) bound on mutual information between two transformations of an image. NCE typically uses randomly sampled negative examples to normalize the objective, but this may often include many uninformative examples either because they are too easy or too hard to discriminate. Taking inspiration from metric learning, we show that choosing semi-hard negatives can yield stronger contrastive representations. To do this, we introduce a family of mutual information estimators that sample negatives conditionally -- in a ringaround each positive. We prove that these estimators remain lower-bounds of mutual information, with higher bias but lower variance than NCE. Experimentally, we find our approach, applied on top of existing models (IR, CMC, and MoCo) improves accuracy by 2-5% absolute points in each case, measured by linear evaluation on four standard image benchmarks. Moreover, we find continued benefits when transferring features to a variety of new image distributions from the Meta-Dataset collection and to a variety of downstream tasks such as object detection, instance segmentation, and key-point detection.

Motivation & Objective

• To address the inefficiency of random negative sampling in contrastive learning, which often includes uninformative examples that are too easy or too hard to discriminate.
• To improve mutual information estimation in contrastive learning by introducing a conditional sampling strategy that focuses on semi-hard negatives.
• To maintain the theoretical guarantee of being a lower bound on mutual information while reducing variance compared to standard NCE.
• To empirically validate that conditional negative sampling enhances representation quality across diverse models and downstream tasks.

Proposed method

• Introduces a family of mutual information estimators that sample negatives conditionally—specifically, in a ring-around each positive sample—based on feature similarity.
• Uses a conditional sampling mechanism that prioritizes negatives that are neither too similar nor too dissimilar to the positive, i.e., semi-hard negatives.
• Proves that the proposed estimators remain valid lower bounds on mutual information, with higher bias but significantly reduced variance compared to standard NCE.
• Applies the method as a plug-in enhancement to existing contrastive learning frameworks such as InfoMin, CMC, and MoCo.
• Employs a temperature-scaled contrastive loss with conditional sampling to improve feature discrimination during training.
• Leverages the ring-around sampling strategy to dynamically adjust negative sampling based on the current feature distribution during optimization.

Experimental results

Research questions

• RQ1 Can conditional negative sampling that focuses on semi-hard examples improve the quality of contrastive visual representations compared to random negative sampling?
• RQ2 Does the proposed conditional sampling method maintain the theoretical lower-bound property of mutual information estimation while reducing variance?
• RQ3 To what extent does the method improve performance across different contrastive learning frameworks (e.g., IR, CMC, MoCo) under linear evaluation?
• RQ4 How does the method generalize when transferring features to new image distributions and diverse downstream tasks?

Key findings

• The proposed conditional negative sampling method improves linear evaluation accuracy by 2–5% absolute points across four standard image benchmarks when applied to existing models.
• The method maintains the lower-bound property of mutual information estimation while reducing variance, leading to more stable and effective training.
• Performance gains are consistent across multiple contrastive learning frameworks, including InfoMin (IR), CMC, and MoCo, demonstrating broad applicability.
• Transfer performance is further enhanced on the Meta-Dataset collection, indicating strong generalization to new image distributions.
• The method yields consistent improvements across diverse downstream tasks, including object detection, instance segmentation, and key-point detection.