[Paper Review] EqCo: Equivalent Rules for Self-supervised Contrastive Learning

Benjin Zhu, Junqiang Huang+3 more|arXiv (Cornell University)|Oct 5, 2020

Domain Adaptation and Few-Shot Learning53 references4 citations

TL;DR

EqCo proposes a theoretically grounded method to stabilize contrastive self-supervised learning by dynamically scaling the margin term in InfoNCE loss according to the number of negative samples, enabling high performance with as few as 16 negatives per query. This eliminates the need for large-batch training or memory banks, achieving ImageNet linear evaluation accuracy comparable to MoCo v2 while using only 16 negative pairs per query.

ABSTRACT

In this paper, we propose EqCo (Equivalent Rules for Contrastive Learning) to make self-supervised learning irrelevant to the number of negative samples in the contrastive learning framework. Inspired by the InfoMax principle, we point that the margin term in contrastive loss needs to be adaptively scaled according to the number of negative pairs in order to keep steady mutual information bound and gradient magnitude. EqCo bridges the performance gap among a wide range of negative sample sizes, so that for the first time, we can use only a few negative pairs (e.g., 16 per query) to perform self-supervised contrastive training on large-scale vision datasets like ImageNet, while with almost no accuracy drop. This is quite a contrast to the widely used large batch training or memory bank mechanism in current practices. Equipped with EqCo, our simplified MoCo (SiMo) achieves comparable accuracy with MoCov2 on ImageNet (linear evaluation protocol) while only involves 16 negative pairs per query instead of 65536, suggesting that large quantities of negative samples is not a critical factor in contrastive learning frameworks.

Motivation & Objective

• To address the empirical and theoretical uncertainty about whether large numbers of negative samples are essential for contrastive self-supervised learning.
• To stabilize the mutual information lower bound and gradient magnitude in contrastive learning when the number of negative samples varies.
• To develop a method that maintains consistent performance across different negative sample sizes, especially for small K.
• To demonstrate that large-scale negative sampling is not a critical factor in contrastive learning when the loss is properly calibrated.
• To provide a theoretically grounded alternative to large-batch training and memory bank mechanisms in contrastive representation learning.

Proposed method

• Introduces an equivalent rule that dynamically scales the margin term in the InfoNCE loss based on the number of negative samples K, ensuring a stable mutual information lower bound.
• Derives a condition (Eq. 5) under which InfoNCE losses with different K values are equivalent in terms of mutual information estimation.
• Proposes a new loss formulation where the margin term is adaptively adjusted via a learnable hyperparameter α, which controls the effective number of negative samples.
• Employs a temperature-scaled contrastive loss with an adaptive margin that maintains consistent gradient magnitude across varying K.
• Validates the method through a toy MI estimation task using correlated Gaussian variables, showing stable MI estimation under EqCo regardless of K.
• Applies EqCo to a simplified MoCo variant (SiMo), replacing the large-batch memory bank with a small K=16, and trains on ImageNet.

Experimental results

Research questions

• RQ1 Is the performance gain from increasing the number of negative samples in contrastive learning fundamentally due to better mutual information estimation, or is it an artifact of unbalanced gradient magnitudes?
• RQ2 Can the mutual information lower bound in contrastive learning be stabilized across different negative sample counts K through proper loss calibration?
• RQ3 Does the need for large-batch training or memory banks in contrastive learning stem from an improperly scaled margin term rather than the quantity of negatives?
• RQ4 Can a small number of negative samples (e.g., K=16) achieve performance comparable to large-batch methods (e.g., K=65536) when the loss is properly calibrated?
• RQ5 Is the InfoNCE loss with adaptive margin (EqCo) generalizable to other contrastive learning frameworks beyond MoCo?

Key findings

• EqCo stabilizes the mutual information lower bound and gradient magnitude across different values of K, making the contrastive loss invariant to the number of negative samples.
• On the ImageNet linear evaluation protocol, SiMo with EqCo and only 16 negative pairs per query achieves 71.8% top-1 accuracy, comparable to MoCo v2 with 65,536 negatives.
• In a toy MI estimation task, EqCo consistently estimates mutual information close to the ground truth across all K values, while standard InfoNCE underestimates MI and becomes increasingly biased as K increases.
• When K ≤ 16, MoCo v2 degrades significantly even with EqCo, indicating that model architecture and training dynamics also influence the effectiveness of the method.
• The performance gain from increasing K beyond a certain point (e.g., from K=256 to K=65536) becomes marginal when using EqCo, suggesting diminishing returns.
• EqCo enables a simplified MoCo variant (SiMo) to match MoCo v2's performance with drastically reduced computational cost and memory usage.