[Paper Review] A Simple Framework for Uncertainty in Contrastive Learning

Mike Wu, Noah D. Goodman|arXiv (Cornell University)|Oct 5, 2020

Domain Adaptation and Few-Shot Learning64 references4 citations

TL;DR

This paper proposes a simple framework, Deep Uncertainty Model (DUM), that learns uncertainty estimates over pretrained contrastive representations by training a deep network to map embeddings to probability distributions, enabling uncertainty-aware representation learning. DUM achieves state-of-the-art performance in anomaly detection (up to 14% absolute improvement) and competitive results in out-of-distribution detection without labels.

ABSTRACT

Contrastive approaches to representation learning have recently shown great promise. In contrast to generative approaches, these contrastive models learn a deterministic encoder with no notion of uncertainty or confidence. In this paper, we introduce a simple approach based on "contrasting distributions" that learns to assign uncertainty for pretrained contrastive representations. In particular, we train a deep network from a representation to a distribution in representation space, whose variance can be used as a measure of confidence. In our experiments, we show that this deep uncertainty model can be used (1) to visually interpret model behavior, (2) to detect new noise in the input to deployed models, (3) to detect anomalies, where we outperform 10 baseline methods across 11 tasks with improvements of up to 14% absolute, and (4) to classify out-of-distribution examples where our fully unsupervised model is competitive with supervised methods.

Motivation & Objective

• To address the lack of uncertainty estimation in deterministic contrastive representation learning models.
• To enable interpretable model behavior analysis through uncertainty-aware representations.
• To detect novel visual corruptions and anomalies in deployed models without retraining.
• To perform unsupervised out-of-distribution detection that rivals supervised methods.
• To provide a plug-in solution compatible with existing contrastive models like SimCLR.

Proposed method

• Train a deep neural network to predict a multivariate Gaussian distribution (mean and variance) from a pretrained contrastive embedding.
• Use a contrastive objective that encourages the model to distinguish the true data distribution from alternative distributions in representation space.
• Optimize the uncertainty model using a contrastive loss that treats the true embedding distribution as positive and perturbed or negative distributions as negatives.
• Leverage the predicted variance as a confidence score for downstream tasks such as anomaly detection and OOD detection.
• Apply the framework to pretrained representations without finetuning the encoder, enabling plug-and-play uncertainty estimation.
• Use the DUM variance as a proxy for representation difficulty, enabling visual interpretability of model confidence.

Experimental results

Research questions

• RQ1 Can a simple, learnable uncertainty head be effectively trained on top of pretrained contrastive representations to estimate confidence?
• RQ2 Can uncertainty estimates improve detection of novel visual corruptions in deployed models?
• RQ3 Can uncertainty-based anomaly detection outperform existing SOTA methods across diverse benchmarks?
• RQ4 Can an unsupervised uncertainty model achieve performance comparable to supervised OOD detection methods?
• RQ5 How does uncertainty correlate with representation quality and model difficulty?

Key findings

• The DUM framework achieves up to 14% absolute improvement in anomaly detection performance across 11 benchmark tasks compared to 10 baseline methods.
• The model detects novel visual noise in images with high accuracy, demonstrating robustness to distribution shift in real-world deployment.
• The DUM-based uncertainty measure enables effective visual interpretability, with high-variance embeddings corresponding to ambiguous or hard-to-embed examples.
• Despite being fully unsupervised, the DUM outperforms several supervised OOD detection methods on standard benchmarks.
• The variance predicted by DUM correlates strongly with representation difficulty, providing a meaningful confidence metric.
• The method is compatible with existing contrastive learning frameworks such as SimCLR and requires no modification to the pretrained encoder.