[Paper Review] Scaling Out-of-Distribution Detection for Real-World Settings

Dan Hendrycks, Steven Basart+6 more|arXiv (Cornell University)|Nov 25, 2019

Anomaly Detection Techniques and Applications35 references179 citations

TL;DR

This paper shows that a simple MaxLogit detector outperforms MSP in large-scale multi-class, multi-label, and anomaly segmentation OOD tasks, and introduces new benchmarks (Species and CAOS) for realistic OOD evaluation.

ABSTRACT

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as detecting novel biological phenomena and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and thousands of classes. To make future work in real-world settings possible, we create new benchmarks for three large-scale settings. To test ImageNet multiclass anomaly detectors, we introduce the Species dataset containing over 700,000 images and over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark for anomaly segmentation by introducing a segmentation benchmark with road anomalies. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks, establishing a simple new baseline for future work.

Motivation & Objective

• Motivate OOD detection in realistic, large-scale settings beyond small-scale benchmarks.
• Create benchmarks for large-scale multi-class (ImageNet-21K), multi-label, and segmentation OOD scenarios.
• Evaluate existing baselines and establish a simple, strong baseline for real-world OOD detection.
• examine whether Vision Transformers pretrained on ImageNet-21K solve OOD detection in large-scale settings.

Proposed method

• Propose MaxLogit: use the negative of the maximum unnormalized logit as the OOD score to avoid class-count bias.
• Construct the Species dataset: a large, disjoint-out-of-distribution set with over 700k images and 1000+ anomalous species to test OOD without train/test overlap.
• Develop and evaluate a multi-label OOD setup on PASCAL VOC and MS-COCO with 20 ImageNet-21K OOD classes, comparing MSP, LogitAvg, and MaxLogit.
• Create the CAOS benchmark combining StreetHazards (simulation-based anomalies) and BDD-Anomaly (real-world anomalies) for anomaly segmentation.
• Compare MaxLogit against baselines (MSP, background, dropout, Reconstruction AE) across StreetHazards and BDD-Anomaly.
• Utilize ImageNet-21K-P representations with ResNet-50, ViT, and Mixer backbones to assess OOD detection performance.

Experimental results

Research questions

• RQ1 Does MSP scale poorly to large-scale OOD detection with thousands of classes?
• RQ2 Is MaxLogit a stronger, universal baseline for large-scale multi-class and multi-label OOD detection?
• RQ3 Do Vision Transformers pretrained on ImageNet-21K inherently solve OOD detection in large-scale settings?
• RQ4 Can we build realistic benchmarks (Species, CAOS) to evaluate OOD under real-world conditions?
• RQ5 How do OOD detectors perform on anomaly segmentation in driving scenes?

Key findings

• MaxLogit consistently outperforms MSP and other baselines across large-scale multi-class, multi-label, and anomaly segmentation tasks.
• Species dataset reveals that Vision Transformers pretrained on ImageNet-21K do not trivially solve OOD detection without careful evaluation to avoid data leakage.
• MaxLogit generalizes well to multi-label settings, outperforming MSP, LogitAvg, and classical detectors.
• CAOS benchmark shows MaxLogit achieves the best performance for pixel-wise anomaly segmentation compared to MSP, background, Dropout, and AE baselines.
• Across StreetHazards and BDD-Anomaly, MaxLogit provides strong, consistent improvements, indicating a robust baseline for real-world OOD detection.