[Paper Review] Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss

Eskil Jörgensen, Christopher Zach+1 more|arXiv (Cornell University)|Jun 19, 2019

Advanced Neural Network Applications35 references65 citations

TL;DR

Introduces SS3D, a single-stage monocular 3D detector that regresses surrogate targets and fits 3D boxes with a differentiable optimizer, trained end-to-end via 3D IoU loss, achieving state-of-the-art monocular KITTI results at real-time speeds (~20 FPS).

ABSTRACT

Three-dimensional object detection from a single view is a challenging task which, if performed with good accuracy, is an important enabler of low-cost mobile robot perception. Previous approaches to this problem suffer either from an overly complex inference engine or from an insufficient detection accuracy. To deal with these issues, we present SS3D, a single-stage monocular 3D object detector. The framework consists of (i) a CNN, which outputs a redundant representation of each relevant object in the image with corresponding uncertainty estimates, and (ii) a 3D bounding box optimizer. We show how modeling heteroscedastic uncertainty improves performance upon our baseline, and furthermore, how back-propagation can be done through the optimizer in order to train the pipeline end-to-end for additional accuracy. Our method achieves SOTA accuracy on monocular 3D object detection, while running at 20 fps in a straightforward implementation. We argue that the SS3D architecture provides a solid framework upon which high performing detection systems can be built, with autonomous driving being the main application in mind.

Motivation & Objective

• Motivate monocular 3D object detection for low-cost perception in robotics/autonomous driving.
• Propose a lightweight single-stage CNN that regresses surrogate 3D representations for accurate 3D box fitting.
• Enable end-to-end training through a non-linear least-squares 3D box optimizer using IoU loss.
• Model and compare uncertainties (homoscedastic vs heteroscedastic) to improve regression reliability.
• Demonstrate state-of-the-art monocular KITTI performance and real-time operation.

Proposed method

• Single-stage CNN (SS3D) outputs 26 surrogate regression targets per detection, plus class scores.
• Non-maximum suppression followed by a parallel non-linear least-squares 3D box fitting per detection.
• 3D box parameters are inferred by minimizing a weighted least-squares objective E(b;y) with weights reflecting per-target uncertainty.
• End-to-end training via backpropagation through the optimizer using 3D IoU loss (IoU3D).
• Three training variants: (i) homoscedastic noise for regression, (ii) heteroscedastic per-output uncertainty, (iii) backprop through optimization with IoU loss.
• Proxy targets include: relative 2D box coords, distance d, observation angle (sin, cos), log-dimension targets, and 3D corner projections as pixel-relative targets.

Experimental results

Research questions

• RQ1 Can a single-stage monocular detector produce competitive 3D detections by regressing surrogate 3D representations and fitting 3D boxes post-detection?
• RQ2 Does modeling per-output uncertainty (homoscedastic/heteroscedastic) improve 3D Box fitting and overall detection accuracy?
• RQ3 Is end-to-end training through a differentiable 3D box optimizer beneficial for monocular 3D object detection on KITTI?
• RQ4 How does SS3D compare to prior monocular and stereo methods in 3D IoU, AP, and ALP on KITTI?
• RQ5 What are the trade-offs between speed and accuracy for SS3D in real-time monocular perception?

Key findings

• SS3D achieves state-of-the-art monocular 3D detection performance on KITTI among published monocular methods.
• Three methods show progressive gains, with Method 3 (end-to-end IoU training) performing best.
• Inference times are around 0.051 seconds per image, enabling ~20 FPS operation.
• Learned uncertainties align with target types (e.g., larger variance for pixel-space targets; distance uncertainty grows with distance).
• End-to-end training through the 3D box optimizer yields further improvements over non-end-to-end training.
• Compared to stereo baselines, SS3D is competitive on KITTI monocular evaluation and substantially faster.