[Paper Review] Complex-YOLO: Real-time 3D Object Detection on Point Clouds

Martín Simón, Stefan Milz+2 more|arXiv (Cornell University)|Mar 16, 2018

Advanced Neural Network Applications24 references78 citations

TL;DR

Complex-YOLO introduces an Euler-Region-Proposal network to estimate 3D oriented boxes directly from LiDAR point clouds in real time, achieving high efficiency and multi-class detection without camera input.

ABSTRACT

Lidar based 3D object detection is inevitable for autonomous driving, because it directly links to environmental understanding and therefore builds the base for prediction and motion planning. The capacity of inferencing highly sparse 3D data in real-time is an ill-posed problem for lots of other application areas besides automated vehicles, e.g. augmented reality, personal robotics or industrial automation. We introduce Complex-YOLO, a state of the art real-time 3D object detection network on point clouds only. In this work, we describe a network that expands YOLOv2, a fast 2D standard object detector for RGB images, by a specific complex regression strategy to estimate multi-class 3D boxes in Cartesian space. Thus, we propose a specific Euler-Region-Proposal Network (E-RPN) to estimate the pose of the object by adding an imaginary and a real fraction to the regression network. This ends up in a closed complex space and avoids singularities, which occur by single angle estimations. The E-RPN supports to generalize well during training. Our experiments on the KITTI benchmark suite show that we outperform current leading methods for 3D object detection specifically in terms of efficiency. We achieve state of the art results for cars, pedestrians and cyclists by being more than five times faster than the fastest competitor. Further, our model is capable of estimating all eight KITTI-classes, including Vans, Trucks or sitting pedestrians simultaneously with high accuracy.

Motivation & Objective

• Motivate real-time 3D object detection for autonomous driving using LiDAR data only.
• Develop a fast, end-to-end network that builds 3D bounding boxes in Cartesian space from bird's-eye-view LiDAR maps.
• Introduce an Euler regression approach (E-RPN) to robustly estimate object orientation without angle singularities.
• Achieve state-of-the-art efficiency while maintaining competitive accuracy on KITTI across multiple classes.

Proposed method

• Preprocess LiDAR point clouds into a single birds-eye-view RGB-map (height, intensity, density) for 80m x 40m coverage.
• Adapt a streamlined YOLOv2-style CNN architecture to operate on the BEV map for single-pass predictions.
• Introduce Euler-Region-Proposal (E-RPN) to regress 3D box parameters (x, y, w, l) and orientation using a complex-number based angle regression (b_phi = arctan2(t_im, t_re)).
• Use three anchor sizes and two orientation directions to cover KITTI object shapes, while predicting five boxes per grid cell with associated scores.
• Combine YOLO-style loss with a novel Euler regression loss to optimize angle predictions in a singularity-free complex space.

Experimental results

Research questions

• RQ1 Can a real-time, LiDAR-only model produce accurate 3D oriented boxes for multiple KITTI classes?
• RQ2 Does embedding angle regression in a complex space (Euler regression) improve orientation robustness and generalization?
• RQ3 What is the trade-off between detection speed and accuracy when using a single BEV map and one forward pass?
• RQ4 Can a single network predict multiple classes simultaneously without camera input while maintaining real-time performance?
• RQ5 How does the proposed method perform on the KITTI benchmark in both BEV and 3D detection tasks?

Key findings

• Achieves real-time performance (>50 fps on a Titan X) while maintaining competitive accuracy on KITTI BEV detection.
• Outperforms leading LiDAR-based methods in efficiency by at least a factor of 5 in BEV detection and by more than 10x in some comparisons.
• Encodes orientation with a complex-angle regression (Euler regression) to avoid angle singularities and improve generalization.
• Predicts eight KITTI classes (including vans, trucks, and sitting pedestrians) from LiDAR input alone, without camera data.
• Provides a single, end-to-end network that processes all bounding boxes in one forward pass, enabling deployment on embedded platforms (e.g., TX2).
• Demonstrates strong BEV and 3D detection performance on CAR, PEDESTRIAN, and CYCLIST categories with competitive AP values.