[Paper Review] Contrastive Training for Improved Out-of-Distribution Detection

Jim Winkens, Rudy Bunel+11 more|arXiv (Cornell University)|Jul 10, 2020

Anomaly Detection Techniques and ApplicationsComputer Science39 references52 citations

TL;DR

This paper uses contrastive training to improve OOD detection without requiring explicit OOD examples, achieving state-of-the-art AUROC on several benchmarks and introducing the Confusion Log Probability (CLP) to quantify task difficulty.

ABSTRACT

Reliable detection of out-of-distribution (OOD) inputs is increasingly understood to be a precondition for deployment of machine learning systems. This paper proposes and investigates the use of contrastive training to boost OOD detection performance. Unlike leading methods for OOD detection, our approach does not require access to examples labeled explicitly as OOD, which can be difficult to collect in practice. We show in extensive experiments that contrastive training significantly helps OOD detection performance on a number of common benchmarks. By introducing and employing the Confusion Log Probability (CLP) score, which quantifies the difficulty of the OOD detection task by capturing the similarity of inlier and outlier datasets, we show that our method especially improves performance in the `near OOD' classes -- a particularly challenging setting for previous methods.

Motivation & Objective

• Motivate robust OOD detection for real-world deployment by learning richer, task-agnostic representations from in-distribution data only.
• Leverage contrastive training to shape feature spaces that preserve sensitivity to in-distribution and potential OOD variations.
• Evaluate the approach across near and far OOD regimes and quantify task difficulty with a new CLP metric.
• Demonstrate practical performance gains over existing methods without requiring OOD labels during training.

Proposed method

• Adopt a SimCLR-style encoder with two projection heads: a classifier head g_φ and a contrastive embedding head h_ν.
• Train using a two-stage objective: first, contrastive loss L_con alone to learn rich representations; second, a joint loss L_con + λ L_class to finalize discrimination.
• Compute the OOD score s(x) by fitting class-conditional Gaussian densities to penultimate activations z, then taking the max over classes as in Eq. (2).
• Use label smoothing to encourage tighter within-class activation clusters, improving density estimation for OOD detection.
• Evaluate OOD performance using AUROC and OOD rank across near, far, and mixed OOD regimes, and introduce Confusion Log Probability (CLP) to quantify task difficulty.

Experimental results

Research questions

• RQ1 Does contrastive training on in-distribution data improve OOD detection without explicit OOD supervision?
• RQ2 How does the proposed method perform across near and far OOD regimes compared to baselines?
• RQ3 Can a robust CLP metric characterize the difficulty of OOD tasks and correlate with detection performance?
• RQ4 What is the impact of combining contrastive training with label smoothing on activation-space separation and OOD scoring?

Key findings

• The proposed method improves OOD detection across near and far OOD settings, achieving 78.3 AUROC on near OOD CIFAR-10 vs CIFAR-100 and 92.9 AUROC on mixed near/far OOD CIFAR-10 vs CIFAR-100.
• In far OOD (CIFAR-10 in vs SVHN out), AUROC reaches 99.5, competitive with state-of-the-art without using labeled OOD data for training or tuning.
• Average AUROC across three dataset pairs reaches 90.2, outperforming prior methods that require outlier data during training.
• Ablation shows combining label smoothing with contrastive training yields the best activation-space shaping and lowest OOD rank variation across runs.
• CLP is effective at illustrating OOD task difficulty, with near OOD scenarios showing higher CLP and thus more challenging detection tasks.
• The method does not rely on explicit outlier data and scales with high-capacity networks, aligning with realistic, unlabelled data settings.