[Paper Review] GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving

Buyu Li, Wanli Ouyang+3 more|arXiv (Cornell University)|Mar 26, 2019

Advanced Neural Network Applications27 references33 citations

TL;DR

GS3D detects 3D objects from a single RGB image by first leveraging reliable 2D detections to generate a coarse 3D cuboid (guidance), then refining it with surface-based features and a classification-style, quality-aware loss to predict accurate 3D boxes.

ABSTRACT

We present an efficient 3D object detection framework based on a single RGB image in the scenario of autonomous driving. Our efforts are put on extracting the underlying 3D information in a 2D image and determining the accurate 3D bounding box of the object without point cloud or stereo data. Leveraging the off-the-shelf 2D object detector, we propose an artful approach to efficiently obtain a coarse cuboid for each predicted 2D box. The coarse cuboid has enough accuracy to guide us to determine the 3D box of the object by refinement. In contrast to previous state-of-the-art methods that only use the features extracted from the 2D bounding box for box refinement, we explore the 3D structure information of the object by employing the visual features of visible surfaces. The new features from surfaces are utilized to eliminate the problem of representation ambiguity brought by only using a 2D bounding box. Moreover, we investigate different methods of 3D box refinement and discover that a classification formulation with quality aware loss has much better performance than regression. Evaluated on the KITTI benchmark, our approach outperforms current state-of-the-art methods for single RGB image based 3D object detection.

Motivation & Objective

• Motivate efficient 3D object detection for autonomous driving using monocular RGB inputs without point clouds or stereo data.
• Propose a two-stage framework that first generates a coarse 3D guidance from 2D detections and scene priors, then refines it using visible-surface features.
• Address representation ambiguity by leveraging 3D structure cues from projected surfaces.
• Show that a classification-based refinement with a quality-aware loss improves 3D box accuracy over regression-based methods.

Proposed method

• Use a Faster R-CNN–based 2D detector with an orientation (observation angle) head to obtain 2D boxes and view directions (alpha).
• Generate a basic 3D cuboid guidance (w_g, h_g, l_g, x_g, y_g, z_g, theta_g) from the 2D box, camera intrinsics, and class-specific size statistics.
• Project the guidance to the image plane and extract features from both the 2D box and the visible surface regions via perspective transformation to capture 3D structure.
• Fuse surface and 2D box features and feed them into a third network (3D subnet) for refinement.
• Formulate 3D box refinement as classification over residual intervals with a quality-aware loss, rather than direct regression, to better reflect prediction quality.
• Explore surface-based features, shift-based classification (per-residual-shift features), and a quality-aware loss to maximize 3D localization accuracy.

Experimental results

Research questions

• RQ1 Can monocular RGB images support accurate 3D object detection for autonomous driving without depth sensors?
• RQ2 Does incorporating visible-surface 3D structure features improve 3D box refinement over 2D-box-only approaches?
• RQ3 Is a classification-based refinement with a quality-aware loss more effective than regression for 3D box refinement?
• RQ4 How far can a monocular framework go toward state-of-the-art results on KITTI using only RGB data and standard priors?

Key findings

• The framework surpasses several prior monocular RGB-based methods on KITTI in 3D AP metrics without using stereo or CAD data.
• Incorporating features from visible surfaces reduces representation ambiguity compared to using only 2D box features.
• A classification formulation with a quality-aware loss for refinement yields better 3D localization than direct regression.
• Surface-aware refinements with shift-based and quality-aware strategies significantly improve 3D box accuracy.