[Paper Review] Positive Unlabeled Contrastive Learning

Anish Acharya, Sujay Sanghavi+5 more|arXiv (Cornell University)|Jun 1, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes Positive Unlabeled Contrastive Learning (puCL), a contrastive representation learning method that leverages labeled positives and unlabeled data to improve downstream PU classification without requiring class prior estimates. It introduces a novel contrastive loss and a PU-specific clustering-based pseudo-labeling strategy, achieving state-of-the-art performance across multiple benchmarks, especially under low-supervision settings.

ABSTRACT

Self-supervised pretraining on unlabeled data followed by supervised fine-tuning on labeled data is a popular paradigm for learning from limited labeled examples. We extend this paradigm to the classical positive unlabeled (PU) setting, where the task is to learn a binary classifier given only a few labeled positive samples, and (often) a large amount of unlabeled samples (which could be positive or negative). We first propose a simple extension of standard infoNCE family of contrastive losses, to the PU setting; and show that this learns superior representations, as compared to existing unsupervised and supervised approaches. We then develop a simple methodology to pseudo-label the unlabeled samples using a new PU-specific clustering scheme; these pseudo-labels can then be used to train the final (positive vs. negative) classifier. Our method handily outperforms state-of-the-art PU methods over several standard PU benchmark datasets, while not requiring a-priori knowledge of any class prior (which is a common assumption in other PU methods). We also provide a simple theoretical analysis that motivates our methods.

Motivation & Objective

• To address the challenge of training robust binary classifiers in the Positive Unlabeled (PU) setting, where only a few labeled positive examples and a large set of unlabeled samples are available.
• To eliminate reliance on the class prior πp, a common assumption in prior PU methods that is often inaccurate or unavailable in practice.
• To improve representation learning in PU settings by adapting contrastive learning to incorporate weak supervision from labeled positives.
• To develop a reliable pseudo-labeling mechanism that leverages clustering on learned representations to generate high-quality negative predictions.
• To achieve state-of-the-art performance in PU learning while remaining robust under limited supervision and without requiring additional side information.

Proposed method

• Proposes a modified contrastive loss, puCL, which extends the standard infoNCE contrastive objective to the PU setting by incorporating labeled positive samples as positive pairs.
• Introduces a self-supervised contrastive pretraining phase using puCL to learn discriminative representations from unlabeled data and a few labeled positives.
• Develops a PU-specific clustering scheme to identify potential negative samples in the unlabeled set based on learned representations.
• Applies pseudo-labeling to the unlabeled set using the clustering results, generating reliable negative predictions for downstream classifier training.
• Trains the final binary classifier via cross-entropy loss on the pseudo-labeled data, combining labeled positives and pseudo-labeled negatives.
• Employs a two-stage pipeline: (1) contrastive pretraining with puCL, and (2) pseudo-labeling followed by supervised fine-tuning.

Experimental results

Research questions

• RQ1 Can contrastive self-supervised learning be effectively adapted to the PU learning setting to improve representation quality without requiring class prior knowledge?
• RQ2 How does the proposed puCL contrastive objective compare to standard self-supervised and supervised contrastive learning in terms of bias-variance trade-off under varying supervision levels?
• RQ3 Can a clustering-based pseudo-labeling strategy effectively identify negative samples in the unlabeled set when representations are well-separated?
• RQ4 Does the proposed method outperform existing PU learning baselines, especially when the number of labeled positive examples is small?
• RQ5 How sensitive is the performance to the true class prior πp, and can the method remain robust when πp is unknown or inaccurately estimated?

Key findings

• puCL significantly outperforms standard self-supervised contrastive learning (ssCL) across all evaluated datasets and supervision levels, with larger gains observed when the number of labeled positives is small.
• The proposed sCL-PU loss, which incorporates class prior, suffers from performance degradation as πp approaches 0.5 and collapses when πp is close to 1, highlighting its sensitivity to prior estimation errors.
• puCL maintains strong performance across varying πp values and remains robust even when the class prior is unknown or inaccurately estimated, unlike prior methods that rely on πp.
• The pseudo-labeling step (puPL) consistently outperforms the nnPU baseline, especially in low-supervision regimes, and prevents model collapse observed in standard cross-entropy training.
• The quality of pseudo-labels is strongly correlated with the separation quality of representations, demonstrating that contrastive pretraining enhances downstream classification performance.
• On CIFAR-I, CIFAR-II, FMNIST-I, and FMNIST-II, puCL achieves state-of-the-art results, outperforming methods that assume access to the true class prior.