[Paper Review] Unbiased Supervised Contrastive Learning

Carlo Alberto Barbano, Benoît Dufumier+3 more|arXiv (Cornell University)|Nov 10, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a novel theoretical framework for supervised contrastive learning that identifies limitations of existing losses (e.g., SupCon) in handling biased data. It introduces $$-SupInfoNCE, a margin-based contrastive loss with improved control over positive-negative sample separation, and FairKL, a debiasing regularization loss. The method achieves state-of-the-art performance on biased vision datasets like CIFAR-10, CIFAR-100, and ImageNet, especially under extreme data biases.

ABSTRACT

Many datasets are biased, namely they contain easy-to-learn features that are highly correlated with the target class only in the dataset but not in the true underlying distribution of the data. For this reason, learning unbiased models from biased data has become a very relevant research topic in the last years. In this work, we tackle the problem of learning representations that are robust to biases. We first present a margin-based theoretical framework that allows us to clarify why recent contrastive losses (InfoNCE, SupCon, etc.) can fail when dealing with biased data. Based on that, we derive a novel formulation of the supervised contrastive loss (epsilon-SupInfoNCE), providing more accurate control of the minimal distance between positive and negative samples. Furthermore, thanks to our theoretical framework, we also propose FairKL, a new debiasing regularization loss, that works well even with extremely biased data. We validate the proposed losses on standard vision datasets including CIFAR10, CIFAR100, and ImageNet, and we assess the debiasing capability of FairKL with epsilon-SupInfoNCE, reaching state-of-the-art performance on a number of biased datasets, including real instances of biases in the wild.

Motivation & Objective

• Address the challenge of learning robust, unbiased representations from biased datasets where spurious features correlate with labels.
• Identify theoretical limitations of existing contrastive losses (e.g., SupCon, InfoNCE) in handling data biases despite low training loss.
• Develop a unified metric learning framework to formally characterize bias and derive new regularization constraints.
• Propose $$-SupInfoNCE, a margin-aware contrastive loss that improves separation between positive and negative samples.
• Introduce FairKL, a novel debiasing regularization loss effective even under extreme data biases, applicable to any contrastive or non-contrastive loss.

Proposed method

• Develop a margin-based theoretical framework to analyze contrastive losses through distance constraints in representation space.
• Reformulate existing losses (InfoNCE, SupCon) within the metric learning framework, revealing hidden non-contrastive constraints in SupCon.
• Propose $$-SupInfoNCE as a generalized, margin-controlled extension of InfoNCE for supervised learning with multiple positives.
• Design FairKL as a regularization loss that enforces equal representation distances for bias-aligned and bias-conflicting samples.
• Apply $$-SupInfoNCE and FairKL jointly during training, optimizing on both the representation space and a projected space when using a projection head.
• Use ablation studies and hyperparameter analysis to tune the trade-off between the main loss and the FairKL regularization term.

Experimental results

Research questions

• RQ1 Why do standard contrastive losses like SupCon fail to learn unbiased representations even when training loss is minimized on biased data?
• RQ2 How can a margin-based formulation improve the robustness of supervised contrastive learning under data bias?
• RQ3 What is the formal metric-based characterization of bias in representation learning, and how can it be leveraged for debiasing?
• RQ4 Can a new regularization loss (FairKL) effectively mitigate bias in highly biased datasets without requiring data reweighting or augmentation?
• RQ5 How does the joint use of $$-SupInfoNCE and FairKL compare to existing state-of-the-art debiasing methods on real-world biased benchmarks?

Key findings

• The $$-SupInfoNCE loss achieves 50.73% accuracy on Corrupted CIFAR-10 with a 5.0% bias ratio, outperforming $$-SupCon and other baselines.
• FairKL regularization improves accuracy by 3.15% on bFFHQ and 1.08% on Corrupted CIFAR-10 (5.0% bias) when using the full regularization term versus mean-only constraints.
• On Biased-MNIST with 99.9% bias, the model with $$-SupInfoNCE and FairKL achieves 95.00% accuracy when $α=0.01$, demonstrating robustness to extreme bias.
• The ablation study confirms that the full FairKL regularization term consistently improves performance across datasets, with gains up to 3.15% on bFFHQ.
• The choice of $α$ (regularization weight) is critical: higher values are needed for stronger biases, with $α=0.1$ optimal for Corrupted CIFAR-10 and bFFHQ.
• Using a projection head reduces performance by up to 3.4% on Corrupted CIFAR-10, suggesting that joint optimization on original and projected spaces may be more effective.