[Paper Review] Detecting Everything in the Open World: Towards Universal Object Detection

Zhenyu Wang, Yali Li+5 more|arXiv (Cornell University)|Mar 21, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes UniDetector, a universal object detection framework that leverages multi-source, heterogeneous datasets and vision-language pre-training to achieve strong zero-shot generalization. By decoupling region proposal generation from classification and applying probability calibration, UniDetector detects over 7,000 categories—surpassing prior methods by 3.5% on novel categories in COCO and 2.6% on LVIS—without fine-tuning, demonstrating state-of-the-art open-world performance with minimal training data.

ABSTRACT

In this paper, we formally address universal object detection, which aims to detect every scene and predict every category. The dependence on human annotations, the limited visual information, and the novel categories in the open world severely restrict the universality of traditional detectors. We propose UniDetector, a universal object detector that has the ability to recognize enormous categories in the open world. The critical points for the universality of UniDetector are: 1) it leverages images of multiple sources and heterogeneous label spaces for training through the alignment of image and text spaces, which guarantees sufficient information for universal representations. 2) it generalizes to the open world easily while keeping the balance between seen and unseen classes, thanks to abundant information from both vision and language modalities. 3) it further promotes the generalization ability to novel categories through our proposed decoupling training manner and probability calibration. These contributions allow UniDetector to detect over 7k categories, the largest measurable category size so far, with only about 500 classes participating in training. Our UniDetector behaves the strong zero-shot generalization ability on large-vocabulary datasets like LVIS, ImageNetBoxes, and VisualGenome - it surpasses the traditional supervised baselines by more than 4\% on average without seeing any corresponding images. On 13 public detection datasets with various scenes, UniDetector also achieves state-of-the-art performance with only a 3\% amount of training data.

Motivation & Objective

• To address the limitations of traditional detectors that rely on fixed, closed-world datasets and fail to generalize to novel, unseen categories.
• To enable universal object detection by training on diverse, multi-source datasets with heterogeneous label spaces, thereby improving coverage and representation learning.
• To enhance zero-shot generalization to unseen categories by leveraging vision-language pre-training and mitigating prediction bias.
• To develop a training paradigm that decouples proposal generation from classification, improving robustness in open-world scenarios.
• To calibrate prediction probabilities to reduce overconfidence in base classes and improve fairness for novel categories.

Proposed method

• UniDetector uses a partitioned training structure to align image and text embeddings across multiple datasets with heterogeneous label spaces, enabling joint learning without label space conflicts.
• It decouples the region proposal generation and RoI classification stages, allowing the proposal network to focus on localization using a class-agnostic localization network (CLN).
• The CLN is trained to produce high-quality, class-agnostic proposals by optimizing for both localization quality and confidence scores, improving generalization to unseen classes.
• Probability calibration is applied by estimating prior probabilities for all categories and reweighting predicted distributions to reduce overconfidence in base classes.
• The framework leverages pre-trained vision-language models (e.g., CLIP) to embed both images and class names, enabling zero-shot inference on novel categories.
• Training is conducted on a combination of COCO, Objects365, and ImageNet with image-level annotations, enabling large-scale, multi-source pre-training.

Figure 1 : Illustration for the universal object detector. It aims to detect every category in every scene and should have the ability to utilize images of multiple sources with heterogeneous label spaces for training and generalize to the open world for inference.

Experimental results

Research questions

• RQ1 Can a universal object detector be trained effectively on multi-source datasets with heterogeneous label spaces while maintaining feature consistency?
• RQ2 How does decoupling region proposal generation from classification improve zero-shot generalization in open-world detection?
• RQ3 To what extent can probability calibration reduce prediction bias and improve performance on unseen categories?
• RQ4 What is the impact of using vision-language pre-training on detecting a large number of novel categories without any fine-tuning?
• RQ5 How does UniDetector compare to existing detectors in terms of performance and data efficiency on large-vocabulary, open-world datasets?

Key findings

• UniDetector achieves 35.2% box AP on novel categories in COCO, surpassing the previous SOTA (RegionCLIP at 31.7%) by 3.5% without any fine-tuning.
• On the LVIS dataset, UniDetector achieves 29.3% box AP and 26.5% mask AP on novel categories, outperforming Detic by 2.6% and 1.6% respectively.
• With only 10% of the training data, UniDetector exceeds OWL-ViT by 1.3% on novel category AP, demonstrating high data efficiency.
• The decoupled training with CLN improves AP by 3.1% over standard RPN and 1.5% over Cascade RPN, showing superior generalization.
• Probability calibration increases novel category AP by over 2% on LVIS, significantly reducing the performance gap between base and novel classes.
• Despite using only one detection dataset for training in some settings, UniDetector achieves state-of-the-art results, indicating strong universality and scalability.

Figure 2 : Overview of UniDetector. It consists of three steps. With the image-text pre-training parameters, UniDetector is trained with images of different sources with multiple label spaces. In this way, it can directly detect in the open world for inference. ’V’ denotes the module to process visu