[Paper Review] Joint stereo 3D object detection and implicit surface reconstruction

Shichao Li, Huang, Xijie+2 more|arXiv (Cornell University)|Nov 25, 2021

Advanced Vision and Imaging86 references4 citations

TL;DR

This paper proposes S-3D-RCNN, a novel two-stage learning framework that jointly performs stereo 3D object detection, accurate SO(3) orientation estimation, and implicit surface reconstruction from RGB stereo images. By introducing Intermediate Geometrical Representations (IGRs) to progressively map image features to 3D object part coordinates and extending them with point-based visible surface modeling, the method achieves state-of-the-art performance on KITTI with new metrics, including improved orientation accuracy and hallucinated shape reconstruction.

ABSTRACT

We present a new learning-based framework S-3D-RCNN that can recover accurate object orientation in SO(3) and simultaneously predict implicit rigid shapes from stereo RGB images. For orientation estimation, in contrast to previous studies that map local appearance to observation angles, we propose a progressive approach by extracting meaningful Intermediate Geometrical Representations (IGRs). This approach features a deep model that transforms perceived intensities from one or two views to object part coordinates to achieve direct egocentric object orientation estimation in the camera coordinate system. To further achieve finer description inside 3D bounding boxes, we investigate the implicit shape estimation problem from stereo images. We model visible object surfaces by designing a point-based representation, augmenting IGRs to explicitly address the unseen surface hallucination problem. Extensive experiments validate the effectiveness of the proposed IGRs, and S-3D-RCNN achieves superior 3D scene understanding performance. We also designed new metrics on the KITTI benchmark for our evaluation of implicit shape estimation.

Motivation & Objective

• To address the challenge of accurate 3D object attribute estimation—especially orientation and shape—from monocular and stereo RGB images in outdoor scenes.
• To overcome the semantic gap between 2D image pixels and 3D geometric quantities by introducing Intermediate Geometrical Representations (IGRs) that bridge appearance and 3D structure.
• To extend single-view orientation estimation to stereo input for improved accuracy and robustness, especially for small or underrepresented objects like pedestrians and cyclists.
• To enable implicit shape reconstruction within 3D bounding boxes by modeling visible surface points and solving the unseen surface hallucination problem.
• To develop new evaluation metrics—AP_MMD and MMDTP—to assess shape retrieval and surface geometry fidelity beyond standard 3D detection metrics.

Proposed method

• The framework uses a two-stage pipeline: first generating 3D object proposals from stereo RGB images, then refining them with IGRs for orientation and shape prediction.
• IGRs are learned by regressing 2D part coordinates from heatmaps and lifting them to 3D object part coordinates in the camera coordinate system.
• A novel point-based representation is introduced to model visible object surface points in the object coordinate system, enabling implicit shape estimation.
• The hallucination module (Ha) generates plausible points for unseen surfaces using the visible point cloud, improving shape completeness.
• A mask head is used to suppress background points, reducing noise in the shape reconstruction process.
• The method is trained with a multi-task loss combining detection, orientation, and shape reconstruction objectives, with a new MMD-based metric for shape fidelity.

Experimental results

Research questions

• RQ1 Can Intermediate Geometrical Representations (IGRs) improve 3D object orientation estimation in stereo settings compared to direct regression?
• RQ2 Does the proposed method generalize to small and underrepresented outdoor objects like pedestrians and cyclists?
• RQ3 Can a point-based representation effectively reconstruct implicit shapes while minimizing hallucination of unseen surfaces?
• RQ4 How does the integration of stereo inputs and IGRs improve orientation accuracy and shape reconstruction compared to single-view baselines?
• RQ5 To what extent is the proposed method robust to noisy 3D bounding box proposals in real-world scenarios?

Key findings

• S-3D-RCNN achieves a 99.58% AOS score on the KITTI validation set under perfect 2D detection (AP2D=100.00), significantly outperforming the baseline (95.22%) and Deep3DBox (98.48%).
• The addition of stereo inputs via PPCs improves AOS by 7.43% on hard samples compared to the single-view Ego-Net, demonstrating the benefit of multi-view aggregation.
• The Ha module reduces MMDTP from 0.027 (without Ha) to 0.0051 in the (0,10m] depth range, proving its effectiveness in hallucinating plausible surface points.
• Mask prediction improves MMDTP by 40% on average, showing that background point suppression enhances shape reconstruction quality.
• Even with increasing translation noise (up to ±0.2m), the method maintains stable performance, indicating robustness to proposal quality variations.
• The new metric AP_MMD shows that Ego-Net++ improves over Disp R-CNN by 12.5% in AP_MMD under IoU > 0.7, validating the effectiveness of the proposed shape retrieval evaluation.