[Paper Review] Contrastive Learning with Hard Negative Samples

Joshua Robinson, Ching-Yao Chuang+2 more|arXiv (Cornell University)|Oct 9, 2020

Domain Adaptation and Few-Shot Learning55 references37 citations

TL;DR

Introduces a tunable unsupervised hard negative sampling method for contrastive learning that emphasizes negatives that are embedding-close to the anchor, with zero overhead and theoretical guarantees.

ABSTRACT

How can you sample good negative examples for contrastive learning? We argue that, as with metric learning, contrastive learning of representations benefits from hard negative samples (i.e., points that are difficult to distinguish from an anchor point). The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use true similarity information. In response, we develop a new family of unsupervised sampling methods for selecting hard negative samples where the user can control the hardness. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.

Motivation & Objective

• Motivate the use of hard negative samples to improve contrastive representation learning.
• Develop an unsupervised, tunable sampling distribution for hard negatives that does not rely on true similarity labels.
• Provide an efficient importance-sampling-based training objective that reweights negatives without extra compute.
• Theoretically characterize the optimal embeddings under hard negative sampling and relate to ball-packing on the hypersphere.
• Empirically demonstrate improved downstream performance across image, graph, and text modalities.

Proposed method

• Propose a negative sampling distribution q_beta^- that biases sampling toward negatives with high current similarity to the anchor: q_beta^-(x^-) ∝ e^{β f(x)^T f(x^-)} p(x^-), conditioned on h(x) ≠ h(x^-).
• Rewrite q_beta to enable practical sampling via a PU-learning decomposition and importance sampling without changing data sampling or adding overhead.
• Derive a hardness-biased objective that uses Monte Carlo estimates of E_{x^- ~ q_beta}[e^{f(x)^T f(x^-)}] and E_{v ~ q_beta^+}[e^{f(x)^T f(v)}], with partition functions Z_beta and Z_beta^+.
• Show that β controls the hardness and that β → ∞ corresponds to a worst-case negative distribution tied to a ball-packing formulation on the hypersphere.
• Provide theoretical results on optimal embeddings under hard negatives and their implications for generalization and simple 1-NN classifiers.

Experimental results

Research questions

• RQ1 Can unsupervised contrastive learning benefit from hard negatives in the absence of true similarity labels?
• RQ2 How should one design a sampling distribution for negatives that emphasizes informative, near-anchor negatives without increasing computational cost?
• RQ3 What are the theoretical properties of representations learned under hard negative sampling, and how do they relate to clustering and ball-packing on the hypersphere?
• RQ4 Does the proposed method lead to improved downstream performance across image, graph, and text modalities?

Key findings

• Hard negative sampling improves downstream task performance on image, graph, and text data across several baselines.
• The proposed q_beta^- with β>0 biases negatives toward those currently similar to the anchor, while maintaining Principle 1 (different labels) approximately via PU-learning decomposition.
• Increasing β yields a spectrum from debiasing to adversarial hard negatives, with β=1 or β=2 often outperforming baselines, and annealing β improving robustness.
• In the infinite-negative-samples limit, optimal embeddings solve a ball-packing problem on the sphere, placing class representatives as far apart as possible.
• Empirical results show notable gains on STL10, CIFAR100, tinyImageNet, and various graph and sentence benchmarks; training can be faster with hard negatives (e.g., STL10 in 60 epochs vs 400 for SimCLR).
• Hard negatives combined with debiasing yield the best linear readout performance on STL10 among the tested configurations.