[Paper Review] Complementary datasets to COCO for object detection

Ali Borji|arXiv (Cornell University)|Jun 23, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces two complementary datasets—COCO_OI and ObjectNet_D—to address the saturation of performance on the COCO object detection benchmark. COCO_OI combines COCO and OpenImages (80 common classes), increasing training data by 80%, while ObjectNet_D provides real-world, diverse-viewpoint images for testing generalization. Models show significant performance drops on these datasets, highlighting limitations in robustness and generalization beyond COCO.

ABSTRACT

For nearly a decade, the COCO dataset has been the central test bed of research in object detection. According to the recent benchmarks, however, it seems that performance on this dataset has started to saturate. One possible reason can be that perhaps it is not large enough for training deep models. To address this limitation, here we introduce two complementary datasets to COCO: i) COCO_OI, composed of images from COCO and OpenImages (from their 80 classes in common) with 1,418,978 training bounding boxes over 380,111 images, and 41,893 validation bounding boxes over 18,299 images, and ii) ObjectNet_D containing objects in daily life situations (originally created for object recognition known as ObjectNet; 29 categories in common with COCO). The latter can be used to test the generalization ability of object detectors. We evaluate some models on these datasets and pinpoint the source of errors. We encourage the community to utilize these datasets for training and testing object detection models. Code and data is available at https://github.com/aliborji/COCO_OI.

Motivation & Objective

• To address performance saturation on COCO by introducing larger, more diverse datasets for training and evaluation.
• To improve model generalization by including real-world, everyday object scenarios with varied viewpoints and backgrounds.
• To create a validation set that diagnoses failure modes in object detectors beyond standard COCO benchmarks.
• To encourage the community to use these datasets to train and test object detection models for improved robustness.

Proposed method

• COCO_OI is constructed by merging COCO and OpenImages using 80 common classes, excluding high-frequency classes (person, car, chair) to prevent data imbalance.
• Images from OpenImages are resized to maintain aspect ratio with a maximum dimension of 640 pixels, ensuring compatibility with COCO's image size distribution.
• ObjectNet_D is created by manually annotating 5,875 images from the ObjectNet dataset, selecting 29 common categories with COCO, one object per image.
• A new validation set is established for both datasets to enable systematic error analysis using TIDE, a diagnostic tool for detection performance.
• Class mappings are defined to align OpenImages and ObjectNet categories with COCO’s 80 classes, enabling cross-dataset evaluation.
• The datasets are released with full licensing details, including restrictions on model fine-tuning for ObjectNet_D.

Experimental results

Research questions

• RQ1 Can combining COCO with OpenImages yield a larger, more diverse training dataset that improves object detection performance?
• RQ2 How do state-of-the-art object detectors generalize to real-world, everyday object scenarios with varied viewpoints and backgrounds?
• RQ3 What are the key failure modes of current detectors when evaluated on COCO_OI and ObjectNet_D compared to COCO?
• RQ4 Can a dedicated validation set from ObjectNet_D effectively diagnose model robustness and generalization limitations?

Key findings

• EfficientDet achieved 51.2 AP on COCO but only 44.0 AP on COCO_OI, indicating a significant performance drop on the extended dataset.
• On ObjectNet_D, EfficientDet's AP dropped to 23.9, with AP_S at 0.0, revealing severe failure on small objects due to large object scale and complex backgrounds.
• DetectoRS showed similar trends, achieving 51.5 AP on COCO_OI and 20.3 AP on ObjectNet_D, confirming that robustness issues are model-agnostic.
• Error analysis using TIDE revealed that models fail more on small objects and in complex scenes, especially in ObjectNet_D, where objects are large and viewed from unusual angles.
• The performance drop on ObjectNet_D—especially the near-zero AP_S—suggests that current detectors struggle with small objects in cluttered, real-world scenes.
• The study confirms that data diversity and real-world variation are critical for improving generalization, and that current benchmarks like COCO may not fully capture detection challenges.