[Paper Review] Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild

Garrick Brazil, Kumar, Abhinav+4 more|arXiv (Cornell University)|Jul 21, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces Omni3D, a large-scale benchmark of 234k images with 3 million 3D object instances across 98 categories, and proposes Cube R-CNN, a unified 3D object detector that generalizes across diverse scenes and camera types using virtual depth to mitigate scale-depth ambiguity. The model achieves state-of-the-art performance, outperforming prior methods by up to 9.5% AP on KITTI and 7.9% on Omni3D, and enables zero-shot generalization to unseen domains like COCO.

ABSTRACT

Recognizing scenes and objects in 3D from a single image is a longstanding goal of computer vision with applications in robotics and AR/VR. For 2D recognition, large datasets and scalable solutions have led to unprecedented advances. In 3D, existing benchmarks are small in size and approaches specialize in few object categories and specific domains, e.g. urban driving scenes. Motivated by the success of 2D recognition, we revisit the task of 3D object detection by introducing a large benchmark, called Omni3D. Omni3D re-purposes and combines existing datasets resulting in 234k images annotated with more than 3 million instances and 98 categories. 3D detection at such scale is challenging due to variations in camera intrinsics and the rich diversity of scene and object types. We propose a model, called Cube R-CNN, designed to generalize across camera and scene types with a unified approach. We show that Cube R-CNN outperforms prior works on the larger Omni3D and existing benchmarks. Finally, we prove that Omni3D is a powerful dataset for 3D object recognition and show that it improves single-dataset performance and can accelerate learning on new smaller datasets via pre-training.

Motivation & Objective

• To address the lack of large-scale, diverse benchmarks for 3D object detection beyond narrow domains like urban driving or indoor scenes.
• To develop a unified 3D object detection model capable of generalizing across diverse camera intrinsics, scene types, and object categories.
• To reduce scale-depth ambiguity in multi-domain 3D detection by introducing virtual depth as a normalization technique.
• To demonstrate that pre-training on Omni3D improves performance on smaller, domain-specific benchmarks.
• To accelerate 3D detection research by releasing a scalable benchmark and efficient evaluation tools.

Proposed method

• Omni3D is constructed by re-purposing and combining existing datasets—SUN RGB-D, ARKitScenes, Hypersim, Objectron, KITTI, and nuScenes—resulting in 234k images and 3M 3D-bounding box annotations across 98 categories.
• A novel, 450× faster algorithm for 3D intersection-over-union (IoU) computation is introduced to enable efficient evaluation on the large-scale benchmark.
• Cube R-CNN is a unified, end-to-end 3D object detector that predicts 3D location, size, and orientation from a single image using a transformer-based backbone and 3D region proposal mechanism.
• Virtual depth is introduced as a normalization strategy: all images are projected into a common virtual camera space with fixed intrinsics to reduce scale-depth ambiguity from varying focal lengths.
• Data augmentation techniques such as image rescaling are enabled via virtual depth, improving generalization and training stability.
• The model is trained end-to-end on the full Omni3D dataset and evaluated on multiple benchmarks, including zero-shot transfer to COCO and unseen domains.

Experimental results

Research questions

• RQ1 Can a single 3D object detector generalize effectively across diverse, real-world scenes with varying camera intrinsics and object categories?
• RQ2 How does pre-training on a large, multi-domain benchmark like Omni3D improve performance on smaller, domain-specific 3D detection benchmarks?
• RQ3 To what extent does virtual depth help mitigate scale-depth ambiguity in monocular 3D detection under variable focal lengths?
• RQ4 Can a unified model outperform specialized models trained on individual datasets like KITTI or SUN RGB-D?
• RQ5 What is the impact of efficient 3D IoU computation on training and evaluation scalability for large-scale 3D detection benchmarks?

Key findings

• Cube R-CNN achieves 9.5% higher AP on the KITTI benchmark compared to the prior state-of-the-art method GUPNet.
• On the Omni3D benchmark, Cube R-CNN outperforms PGD by 7.9% AP, demonstrating strong generalization across diverse scenes.
• Pre-training on Omni3D improves single-dataset performance by up to 5.3% AP on urban benchmarks and 3.8% on indoor benchmarks.
• The proposed 3D IoU computation algorithm is 450× faster than prior methods, enabling efficient training and evaluation on large-scale datasets.
• Cube R-CNN generalizes effectively to unseen domains, achieving zero-shot detection on COCO images with unknown camera intrinsics, particularly for indoor scenes.
• Performance degrades on long-tail categories in Omni3D, with AP3D dropping from 23.3% (top 50 categories) to 14.1% (full 98 categories), highlighting challenges in few-shot 3D recognition.