[Paper Review] Discriminative out-of-distribution detection for semantic segmentation

Petra Bevandić, Ivan Krešo+2 more|arXiv (Cornell University)|Aug 23, 2018

Advanced Image and Video Retrieval Techniques23 references45 citations

TL;DR

The paper introduces a dedicated OOD detector trained to distinguish in-distribution road-driving pixels from a broad background (ILSVRC) to identify OOD pixels in semantic segmentation, showing improved OOD detection performance over prior methods.

ABSTRACT

Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes. However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input. These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive. We propose to address this issue by discriminative detection of OOD pixels in input data. Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task. Instead, we train a dedicated OOD model which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world. We perform our experiments on high resolution natural images in a dense prediction setup. We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset. We evaluate our approach on WildDash test, which is currently the only public test dataset that includes out-of-distribution images. The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin.

Motivation & Objective

• Motivate robust OOD detection in semantic segmentation for real-world driving scenarios where closed-world assumptions fail.
• Propose a discriminative, fully convolutional OOD detector trained independently from the primary segmentation model.
• Use a large background dataset (ILSVRC) to approximate the visual world diversity and separate OOD from in-distribution data.
• Evaluate the method against several baselines and on road-driving and WildDash datasets to demonstrate improved OOD detection performance.

Proposed method

• Train a dedicated OOD detector to perform binary pixel-wise classification between in-distribution (ID) pixels from road-driving datasets and out-of-distribution (OOD) pixels from ILSVRC.
• Use DenseNet-121 based feature extractor initialized with ILSVRC pretraining, and fine-tune only selected layers (DB4 and T3) plus the OOD head to discriminate road-driving scenes from the general background.
• Form mixed batches of 512x512 crops from ID and OOD sources, oversampling ID data to balance training, and label all ILSVRC pixels as OOD while road-driving pixels are ID.
• Train the OOD model alongside primary segmentation models (Cityscapes, Vistas, WildDash) to compare dense OOD detection performance against max-softmax, ODIN, and trained confidence baselines.
• Evaluate OOD detection using average precision on WildDash test, with two variants: considering all negative images and excluding altered negatives.

Experimental results

Research questions

• RQ1 Can a discriminative, task-agnostic OOD detector trained on a large background dataset (ILSVRC) reliably identify OOD pixels in semantic segmentation tasks?
• RQ2 Does separating OOD detection from the primary segmentation model yield higher OOD detection performance than jointly adapting uncertainty strategies to the primary model?
• RQ3 How does training the OOD detector with different in-distribution datasets (Cityscapes, Cityscapes+WildDash, Vistas) affect OOD detection performance on WildDash and other datasets?
• RQ4 What is the impact of using a background dataset like ILSVRC for OOD detection on various road-driving and indoor/outdoor scenes?
• RQ5 How do conventional image-wide OOD methods (max-softmax, ODIN, trained confidence) perform when adapted to dense prediction versus the proposed discriminative approach?

Key findings

• The discriminative OOD detector trained with ILSVRC as background significantly outperforms max-softmax and other adapted baselines on WildDash OOD detection AP.
• Training the OOD detector with diverse background data (especially including WildDash val) yields the best OOD detection performance, indicating the benefit of including challenging examples in training.
• Models trained on Vistas as ID data achieve strong OOD detection, while Cityscapes alone may lead to overfitting and poorer generalization to WildDash.
• Joint training on ILSVRC and road-driving datasets produces the best OOD detection performance, with substantial AP gains over baselines across various test setups.
• The approach demonstrates the importance of a comprehensive background distribution and pixel-level OOD labeling to improve epistemic uncertainty estimation in dense prediction.