[Paper Review] A Benchmark for Anomaly Segmentation.

Dan Hendrycks, Steven Basart+4 more|arXiv (Cornell University)|Nov 25, 2019

Anomaly Detection Techniques and Applications34 references50 citations

TL;DR

This paper introduces the Combined Anomalous Object Segmentation (CAOS) benchmark for large-scale anomaly segmentation, combining real and simulated driving images to enable realistic, diverse anomaly detection. It proposes novel out-of-distribution detectors for multi-class and multi-label settings, significantly advancing large-scale anomaly segmentation beyond whole-image anomaly detection.

ABSTRACT

Detecting out-of-distribution examples is important for safety-critical machine learning applications such as self-driving vehicles. However, existing research mainly focuses on small-scale images where the whole image is considered anomalous. We propose to segment only the anomalous regions within an image, and hence we introduce the Combined Anomalous Object Segmentation benchmark for the more realistic task of large-scale anomaly segmentation. Our benchmark combines two novel datasets for anomaly segmentation that incorporate both realism and anomaly diversity. Using both real images and those from a simulated driving environment, we ensure the background context and a wide variety of anomalous objects are naturally integrated, unlike before. Additionally, we improve out-of-distribution detectors on large-scale multi-class datasets and introduce detectors for the previously unexplored setting of multi-label out-of-distribution detection. These novel baselines along with our anomaly segmentation benchmark open the door to further research in large-scale out-of-distribution detection and segmentation.

Motivation & Objective

• To address the gap in large-scale, instance-level anomaly detection by focusing on segmenting only anomalous regions rather than classifying entire images as anomalous.
• To create a realistic benchmark that integrates diverse anomalous objects into natural background contexts using real and simulated driving data.
• To improve out-of-distribution detection performance on large-scale multi-class datasets with novel detection methods.
• To introduce and evaluate detectors for the previously unexplored setting of multi-label out-of-distribution detection.

Proposed method

• Constructs a new benchmark, CAOS, by combining two novel datasets: one from real-world driving images and another from a simulated driving environment.
• Integrates realistic background scenes with a wide variety of anomalous objects to ensure contextual and anomaly diversity.
• Proposes new out-of-distribution detection models tailored for large-scale multi-class and multi-label anomaly segmentation tasks.
• Adapts and improves existing OOD detection techniques for instance-level segmentation, focusing on localization of anomalous regions.
• Uses both real and synthetic data to ensure robustness and generalization across diverse anomaly types and environmental conditions.
• Employs segmentation heads that predict pixel-wise anomaly scores, enabling precise localization of anomalous objects within images.

Experimental results

Research questions

• RQ1 How can large-scale anomaly segmentation be realistically benchmarked using diverse, contextually integrated anomalous objects?
• RQ2 What improvements can be achieved in out-of-distribution detection when focusing on segmenting only anomalous regions rather than classifying entire images?
• RQ3 How do existing OOD detection methods perform when adapted to multi-class and multi-label anomaly segmentation settings?
• RQ4 Can simulated data effectively support training and evaluation of anomaly segmentation models in realistic driving scenarios?

Key findings

• The CAOS benchmark enables more realistic evaluation of anomaly segmentation models by combining real and simulated driving data with diverse, naturally integrated anomalies.
• The proposed detectors achieve improved performance on large-scale multi-class anomaly segmentation benchmarks compared to prior methods.
• The study introduces the first effective approach for multi-label out-of-distribution detection in segmentation tasks, expanding the scope of OOD research.
• The integration of simulated data enhances model generalization and enables training with a broader range of anomaly types and scene contexts.