[Paper Review] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection

Jingkang Yang, Pengyun Wang+14 more|arXiv (Cornell University)|Oct 13, 2022

Anomaly Detection Techniques and ApplicationsComputer Science86 citations

TL;DR

OpenOOD provides a unified, generalized benchmark and codebase for evaluating anomaly detection, open set recognition, and out-of-distribution detection methods, revealing trends across 9 benchmarks and 35 methods.

ABSTRACT

Out-of-distribution (OOD) detection is vital to safety-critical machine learning applications and has thus been extensively studied, with a plethora of methods developed in the literature. However, the field currently lacks a unified, strictly formulated, and comprehensive benchmark, which often results in unfair comparisons and inconclusive results. From the problem setting perspective, OOD detection is closely related to neighboring fields including anomaly detection (AD), open set recognition (OSR), and model uncertainty, since methods developed for one domain are often applicable to each other. To help the community to improve the evaluation and advance, we build a unified, well-structured codebase called OpenOOD, which implements over 30 methods developed in relevant fields and provides a comprehensive benchmark under the recently proposed generalized OOD detection framework. With a comprehensive comparison of these methods, we are gratified that the field has progressed significantly over the past few years, where both preprocessing methods and the orthogonal post-hoc methods show strong potential.

Motivation & Objective

• Create a unified, well-structured benchmark for generalized OOD detection to enable fair comparisons across related tasks (AD, OSR, OOD).
• Provide a comprehensive, easily extensible codebase (OpenOOD) implementing 35 methods from related fields for fair evaluation.
• Design 9 benchmarks with near-OOD and far-OOD distinctions to enable detailed analysis of ID/OOD separability.
• Offer empirical insights on preprocessing, data usage, and post-hoc vs training-based methods to guide future research.

Proposed method

• Build a unified OpenOOD codebase integrating 35 methods from AD, OSR, OOD detection, and model uncertainty with a common experimental framework.
• Provide 9 benchmarks spanning AD, OSR, and OOD detection under a generalized OOD detection framework, including near-OOD and far-OOD scenarios.
• Standardize preprocessing, model backbones, and training/testing protocols to ensure fair comparisons across methods.
• Evaluate methods using common metrics (FPR@95, AUROC, AUPR) with a primary focus on AUROC.
• Analyze results to identify trends such as the effectiveness of simple preprocessing, the role of extra data, and the impact of post-hoc vs training-based approaches.

Experimental results

Research questions

• RQ1 How do diverse OOD-related methods perform under a unified generalized OOD benchmark across AD, OSR, and OOD tasks?
• RQ2 Do simple preprocessing or post-hoc methods achieve competitive performance compared to training-based approaches in generalized OOD detection?
• RQ3 What is the impact of using extra outlier data on OOD detection performance, and how do near-OOD and far-OOD scenarios differ in practice?
• RQ4 How do OSR benchmarks align with OOD detection benchmarks in terms of method effectiveness and ID accuracy?

Key findings

• Post-hoc and preprocessing methods often outperform more heavily trained approaches in the generalized benchmark.
• Extra outlier data does not consistently improve performance, and its benefit is dataset-dependent.
• Data augmentation and model uncertainty methods (e.g., PixMix, Mixup, ensembles) show strong performance across benchmarks.
• ImageNet-based evaluations tend to show higher AUROC scores than CIFAR-based ones, indicating dataset complexity effects.
• OSR results tend to align with near-OOD performance, suggesting shared challenges between OSR and OOD detection.
• The OpenOOD framework enables fair comparisons and highlights that simple, practical techniques can yield substantial gains.