[Paper Review] Unknown-Aware Object Detection: Learning What You Don't Know from Videos in the Wild

Xuefeng Du, Xin Wang+2 more|arXiv (Cornell University)|Mar 8, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes STUD, a novel unknown-aware object detection framework that distills diverse out-of-distribution (OOD) objects from videos in the wild using spatial-temporal unknown distillation and energy-based uncertainty regularization. By leveraging unlabeled video data to regularize decision boundaries without human-annotated OOD data, STUD achieves state-of-the-art OOD detection performance, reducing FPR95 by over 10% on BDD100K while preserving in-distribution detection accuracy.

ABSTRACT

Building reliable object detectors that can detect out-of-distribution (OOD) objects is critical yet underexplored. One of the key challenges is that models lack supervision signals from unknown data, producing overconfident predictions on OOD objects. We propose a new unknown-aware object detection framework through Spatial-Temporal Unknown Distillation (STUD), which distills unknown objects from videos in the wild and meaningfully regularizes the model's decision boundary. STUD first identifies the unknown candidate object proposals in the spatial dimension, and then aggregates the candidates across multiple video frames to form a diverse set of unknown objects near the decision boundary. Alongside, we employ an energy-based uncertainty regularization loss, which contrastively shapes the uncertainty space between the in-distribution and distilled unknown objects. STUD establishes the state-of-the-art performance on OOD detection tasks for object detection, reducing the FPR95 score by over 10% compared to the previous best method. Code is available at https://github.com/deeplearning-wisc/stud.

Motivation & Objective

• To address the critical yet underexplored challenge of out-of-distribution (OOD) object detection in real-world scenarios where models often produce overconfident predictions on unseen objects.
• To develop a method that enables object detectors to learn uncertainty about unknown objects without requiring costly human annotations of OOD data.
• To improve decision boundary generalization between in-distribution and OOD objects using self-supervised distillation from natural video data.
• To establish a new paradigm for OOD detection in object detection by exploiting spatial and temporal diversity in videos.
• To preserve high in-distribution detection accuracy while significantly enhancing OOD detection performance.

Proposed method

• Spatial-Temporal Unknown Distillation (STUD) identifies unknown object candidates in reference frames using an OOD measurement, then linearly combines them in feature space with dissimilarity-based weighting to form diverse distilled unknowns.
• The method aggregates distilled unknowns across multiple video frames to capture temporal diversity and enrich the distribution of OOD-like samples.
• An energy-based uncertainty regularization loss is introduced to contrastively shape the uncertainty space, pushing higher scores for in-distribution objects and lower scores for distilled unknowns.
• The framework jointly optimizes object detection and OOD detection by training a shared backbone with an uncertainty regularization branch that guides the decision boundary toward greater conservatism.
• The approach avoids synthetic OOD generation (e.g., via GANs) or negative proposal usage, which are shown to be suboptimal in ablation studies.
• The method is trained end-to-end on standard video datasets like BDD100K and Youtube-VIS, using only standard ID annotations and natural video content for OOD signal extraction.

Experimental results

Research questions

• RQ1 Can out-of-distribution objects be effectively distilled from unannotated videos in the wild to improve OOD detection without human-annotated OOD data?
• RQ2 How can spatial and temporal diversity in video data be leveraged to construct meaningful, representative unknown object proxies for model regularization?
• RQ3 Can energy-based uncertainty regularization improve the decision boundary between in-distribution and out-of-distribution objects in object detection?
• RQ4 Does distilling unknowns from real videos outperform synthetic OOD generation or negative proposal-based methods in OOD detection performance?
• RQ5 To what extent can OOD detection be improved without degrading in-distribution detection accuracy?

Key findings

• STUD reduces the FPR95 score by 10.88% on the BDD100K dataset compared to the previous state-of-the-art method, demonstrating superior OOD detection performance.
• The method preserves high in-distribution detection accuracy, showing that OOD generalization does not come at the cost of ID performance.
• Qualitative results show that STUD reduces false positive predictions on OOD objects and lowers confidence scores for such false positives compared to vanilla detectors.
• Ablation studies confirm that spatial-temporal distillation and energy-based regularization are both essential, with the proposed method outperforming GAN-based synthesis and negative proposal baselines.
• The framework effectively leverages natural video data to extract meaningful OOD signals, eliminating the need for expensive OOD annotation.
• The energy-based uncertainty regularization successfully shapes the uncertainty surface, enabling better separation between ID and OOD predictions during inference.