[Paper Review] Exploring the Limits of Out-of-Distribution Detection

Stanislav Fort, Jie Ren+1 more|arXiv (Cornell University)|Jun 6, 2021

Anomaly Detection Techniques and ApplicationsComputer Science53 references107 citations

TL;DR

The paper shows that large-scale pre-trained transformers, especially Vision Transformers (ViT), significantly improve near-OOD detection across vision and genomics, and that few-shot outlier exposure and zero-shot multi-modal cues further boost performance.

ABSTRACT

Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-trained transformers can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% to 77% using transformers and unsupervised pre-training. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained transformers are particularly well-suited for outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained transformers such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information without any accompanying images, and show that this outperforms previous SOTA on standard vision OOD benchmark tasks.

Motivation & Objective

• Demonstrate that large-scale pre-trained transformers improve near-OOD detection across modalities.
• Quantify OOD detection gains on CIFAR-100 vs CIFAR-10 and genomics benchmarks.
• Assess the effect of fine-tuning, different architectures, and self-supervised pre-training on OOD performance.
• Explore few-shot outlier exposure as a practical method to boost OOD detection.
• Investigate zero-shot OOD detection using multi-modal models like CLIP by leveraging outlier class names.

Proposed method

• Fine-tune Vision Transformers (ViT) pre-trained on ImageNet-21k on CIFAR-10/CIFAR-100 and evaluate MSP and Mahalanobis distance for OOD detection.
• Compare ViT with BiT (ResNet-based) and MLP-Mixer architectures to assess architecture benefits.
• Use outlier exposure to train a simple classifier (linear for supervised-pretrained models, shallow MLP for unsupervised-pretrained models) with in-distribution and few-shot outliers.
• Vary the number of outlier examples per class (1–10, and more) to study few-shot OOD improvement.
• Apply CLIP-style zero-shot OOD detection by using names of outlier classes as candidate text labels and measuring OOD scores from image-text alignment.

Experimental results

Research questions

• RQ1 How much do large-scale pre-trained transformers improve near-OOD detection compared to state-of-the-art baselines?
• RQ2 What is the impact of fine-tuning on in-distribution data versus using pre-trained features alone for OOD detection?
• RQ3 How does few-shot outlier exposure influence AUROC in CIFAR-100 vs CIFAR-10 and CIFAR-10 vs CIFAR-100 tasks?
• RQ4 Can multi-modal, zero-shot cues from models like CLIP improve OOD detection without labeled OOD images?
• RQ5 How does unsupervised pre-training (e.g., DINO) compare to supervised pre-training for OOD detection?

Key findings

• ViT fine-tuned on CIFAR-100 with Mahalanobis distance achieves AUROC of 96% for CIFAR-100 vs CIFAR-10, surpassing the prior SOTA of 85%.
• Pre-trained ViT (ImageNet-21k) outperforming BiT and MLP-Mixer baselines on near-OOD tasks.
• Few-shot outlier exposure with 1–10 labeled OOD examples per class yields ~99% AUROC on CIFAR-100 vs CIFAR-10 when using fine-tuned ViT features.
• Genomics OOD detection improves from 66% to 77% AUROC with pre-trained+fine-tuned transformers (BERT) using MSP and Mahalanobis distance.
• Zero-shot OOD detection with CLIP using only outlier class names achieves AUROC of 94.8% on CIFAR-100 vs CIFAR-10 and near-perfect AUROC on some far-OOD tasks (e.g., 99.6%/99.9%).
• Genomics OOD with pre-trained+fine-tuned transformers yields higher AUROC (77.49% Mahalanobis, 73.53% MSP) and better in-distribution accuracy (89.84%).