[Paper Review] Unsupervised Recognition of Unknown Objects for Open-World Object Detection

Ruohuan Fang, Guansong Pang+3 more|arXiv (Cornell University)|Aug 31, 2023

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes MEPU, a novel unsupervised framework for open-world object detection that mitigates label bias by using a reconstruction error-based Weibull model (REW) to distinguish true unknown objects from background regions in pseudo-labeled proposals. By modeling feature frequency distributions via autoencoders and Weibull statistics, MEPU achieves state-of-the-art performance in detecting unknown objects while maintaining strong known object detection accuracy on MS COCO, LVIS, and Objects365.

ABSTRACT

Open-World Object Detection (OWOD) extends object detection problem to a realistic and dynamic scenario, where a detection model is required to be capable of detecting both known and unknown objects and incrementally learning newly introduced knowledge. Current OWOD models, such as ORE and OW-DETR, focus on pseudo-labeling regions with high objectness scores as unknowns, whose performance relies heavily on the supervision of known objects. While they can detect the unknowns that exhibit similar features to the known objects, they suffer from a severe label bias problem that they tend to detect all regions (including unknown object regions) that are dissimilar to the known objects as part of the background. To eliminate the label bias, this paper proposes a novel approach that learns an unsupervised discriminative model to recognize true unknown objects from raw pseudo labels generated by unsupervised region proposal methods. The resulting model can be further refined by a classification-free self-training method which iteratively extends pseudo unknown objects to the unlabeled regions. Experimental results show that our method 1) significantly outperforms the prior SOTA in detecting unknown objects while maintaining competitive performance of detecting known object classes on the MS COCO dataset, and 2) achieves better generalization ability on the LVIS and Objects365 datasets.

Motivation & Objective

• Address the severe label bias problem in existing open-world object detection (OWOD) methods that misclassify dissimilar regions as background due to reliance on known object supervision.
• Enable unsupervised recognition of true unknown objects from raw pseudo-labeled regions generated by unsupervised region proposal methods.
• Improve generalization and detection performance for unknown objects across diverse datasets like LVIS and Objects365 without requiring unknown class annotations.
• Develop a self-training pipeline that iteratively extends detection to unlabeled regions using a classification-free refinement strategy.
• Overcome limitations of existing reconstruction-based OOD detection methods by introducing adaptive thresholding via Weibull modeling of reconstruction errors.

Proposed method

• Propose a reconstruction error-based Weibull model (REW) that uses autoencoders to encode region features and models reconstruction errors with a Weibull distribution to distinguish foreground (unknown objects) from background (uniform textures).
• Train the REW module using a separate backbone to preserve semantic expressiveness of dense features, avoiding interference from downstream detection tasks.
• Assign soft labels to pseudo-labeled regions based on the likelihood of being true unknown objects, derived from the Weibull model's reconstruction error distribution.
• Integrate REW with a detection head that uses these soft labels to refine unknown object predictions and reduce false positives from background regions.
• Apply a classification-free self-training strategy (ROLNet) that extends detection to unlabeled regions by leveraging high-scoring pseudo-labels from the REW module.
• Use a joint training scheme where the autoencoder and detection model are trained end-to-end, though ablation shows performance drops when using online training due to feature bias.

Experimental results

Research questions

• RQ1 Can unsupervised modeling of feature frequency distributions effectively distinguish true unknown objects from background regions without relying on known object supervision?
• RQ2 How does the REW module’s Weibull-based reconstruction error modeling improve detection of unknown objects compared to objectness-score-based pseudo-labeling in existing OWOD methods?
• RQ3 To what extent does the self-training mechanism (ROLNet) enhance detection of previously unseen unknown objects in unlabeled regions?
• RQ4 Does the proposed method generalize better than prior SOTA approaches on cross-dataset evaluation, particularly on LVIS and Objects365?
• RQ5 What is the impact of using a separate reconstruction backbone versus online training on the overall detection performance and feature quality?

Key findings

• MEPU achieves a 54.8% unknown object R@100 on the S-OWODB Task-1 benchmark, significantly outperforming prior SOTA methods that rely on objectness scores and suffer from label bias.
• The REW module reduces false positives on background regions by modeling feature frequency distributions, assigning low scores (0.0–0.3) to uniform textures like sky and walls, while giving high scores (0.7–1.0) to true unknown objects.
• On the LVIS and Objects365 datasets, MEPU demonstrates better generalization ability than existing OWOD models, indicating robustness across diverse and open-ended object categories.
• Ablation studies show that using a separate reconstruction backbone (MEPU-FS) yields better performance than online training (−0.3 mAP and −4.1 R@100 drop), confirming the importance of preserving semantic expressiveness in feature learning.
• Self-training with ROLNet improves detection of unknown objects, but replacing OLN with RPN-based pseudo-labeling leads to a 6.1-point drop in R@100, indicating that the OLN module is critical for maintaining detection quality.
• Failure cases reveal limitations in complex backgrounds or partial object detection, where the model sometimes misclassifies textured backgrounds or object parts as unknowns due to lack of semantic supervision.