[Paper Review] NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection

Chenfeng Xu, BoRui Wu+9 more|arXiv (Cornell University)|Jul 27, 2023

Advanced Neural Network Applications4 citations

TL;DR

NeRF-Det proposes a geometry-aware volumetric representation for multi-view 3D object detection using only RGB images by jointly training a NeRF branch with a detection head via a shared MLP. It achieves state-of-the-art performance, improving mAP by 3.9 and 3.1 on ScanNet and ARKITScenes respectively, without per-scene NeRF optimization, while enabling generalizable novel view synthesis and depth estimation.

ABSTRACT

We present NeRF-Det, a novel method for indoor 3D detection with posed RGB images as input. Unlike existing indoor 3D detection methods that struggle to model scene geometry, our method makes novel use of NeRF in an end-to-end manner to explicitly estimate 3D geometry, thereby improving 3D detection performance. Specifically, to avoid the significant extra latency associated with per-scene optimization of NeRF, we introduce sufficient geometry priors to enhance the generalizability of NeRF-MLP. Furthermore, we subtly connect the detection and NeRF branches through a shared MLP, enabling an efficient adaptation of NeRF to detection and yielding geometry-aware volumetric representations for 3D detection. Our method outperforms state-of-the-arts by 3.9 mAP and 3.1 mAP on the ScanNet and ARKITScenes benchmarks, respectively. We provide extensive analysis to shed light on how NeRF-Det works. As a result of our joint-training design, NeRF-Det is able to generalize well to unseen scenes for object detection, view synthesis, and depth estimation tasks without requiring per-scene optimization. Code is available at \url{https://github.com/facebookresearch/NeRF-Det}.

Motivation & Objective

• Address the challenge of 3D object detection in indoor scenes using only RGB images, without depth sensors.
• Overcome the limitations of implicit geometry modeling in existing methods that rely on ambiguous 3D volume representations.
• Integrate NeRF's strong geometric inductive bias into 3D detection without incurring per-scene optimization latency.
• Enable generalization to unseen scenes for detection, novel view synthesis, and depth estimation without retraining per scene.
• Improve detection performance by leveraging multi-view consistency through a joint training paradigm with NeRF and detection branches.

Proposed method

• Introduce a shared MLP between NeRF and detection branches to enable gradient flow from NeRF to image features, improving geometry-aware representation.
• Sample features along rays from high-resolution image feature maps instead of low-resolution volume features to preserve spatial detail.
• Augment image features with spatial statistics (mean and variance) and color information as geometry priors to enhance NeRF generalization.
• Transform the predicted density field into an opacity field and modulate volume features, suppressing empty space and improving detection accuracy.
• Train the NeRF and detection branches end-to-end with a photo-metric loss for multi-view consistency and optional depth supervision.
• Remove the NeRF branch at inference, minimizing overhead while retaining geometry-aware features for detection.

Experimental results

Research questions

• RQ1 Can NeRF be effectively integrated into 3D object detection without per-scene optimization to maintain efficiency?
• RQ2 How does joint training of NeRF and detection branches improve geometry modeling and detection performance?
• RQ3 To what extent do image-level geometry priors (e.g., mean, variance, color) enhance NeRF generalization to unseen scenes?
• RQ4 Does the NeRF branch generalize to novel view synthesis and depth estimation without per-scene fine-tuning?
• RQ5 What is the relative contribution of photo-metric loss and depth supervision to NeRF-based geometry learning in detection?

Key findings

• NeRF-Det achieves 50.1 mAP@0.25 and 24.4 mAP@0.50 on ScanNet, outperforming SOTA by 3.9 mAP.
• On ARKITScenes, NeRF-Det achieves 50.0 mAP@0.25 and 24.2 mAP@0.50, improving SOTA by 3.1 mAP.
• Using image feature sampling instead of volume feature sampling improves mAP by 0.7 and PSNR by 1.58 for novel view synthesis.
• Incorporating variance and color features boosts mAP by 0.7 and 0.6 for mAP@0.25 and mAP@0.50 respectively compared to mean-only features.
• The NeRF branch generalizes well to unseen scenes, achieving 20.51 PSNR in novel view synthesis and 0.756 RMSE in depth estimation without per-scene optimization.
• Removing the detection branch slightly improves NeRF performance (PSNR 20.94 vs 20.51), suggesting detection may suppress low-level details useful for NeRF.