[Paper Review] VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection

Yin Zhou, Oncel Tuzel|arXiv (Cornell University)|Nov 17, 2017

Advanced Neural Network Applications39 references314 citations

TL;DR

VoxelNet introduces an end-to-end trainable framework that operates directly on sparse 3D LiDAR point clouds by voxelizing space, applying voxel feature encoding, and using an RPN to predict 3D bounding boxes, achieving state-of-the-art results on KITTI without hand-crafted features.

ABSTRACT

Accurate detection of objects in 3D point clouds is a central problem in many applications, such as autonomous navigation, housekeeping robots, and augmented/virtual reality. To interface a highly sparse LiDAR point cloud with a region proposal network (RPN), most existing efforts have focused on hand-crafted feature representations, for example, a bird's eye view projection. In this work, we remove the need of manual feature engineering for 3D point clouds and propose VoxelNet, a generic 3D detection network that unifies feature extraction and bounding box prediction into a single stage, end-to-end trainable deep network. Specifically, VoxelNet divides a point cloud into equally spaced 3D voxels and transforms a group of points within each voxel into a unified feature representation through the newly introduced voxel feature encoding (VFE) layer. In this way, the point cloud is encoded as a descriptive volumetric representation, which is then connected to a RPN to generate detections. Experiments on the KITTI car detection benchmark show that VoxelNet outperforms the state-of-the-art LiDAR based 3D detection methods by a large margin. Furthermore, our network learns an effective discriminative representation of objects with various geometries, leading to encouraging results in 3D detection of pedestrians and cyclists, based on only LiDAR.

Motivation & Objective

• Motivate LiDAR-based 3D object detection without hand-crafted features.
• Propose a unified end-to-end architecture that learns point-wise and voxel-level representations.
• Eliminate the need for manual feature engineering by introducing voxel feature encoding layers.
• Demonstrate state-of-the-art performance on KITTI Car, Pedestrian, and Cyclist detection using LiDAR alone.

Proposed method

• Partition the 3D space into a regular voxel grid and group points by voxel.
• Introduce voxel feature encoding (VFE) layers to learn per-voxel features by combining point-wise and locally aggregated information.
• Stack multiple VFE layers to capture intra-voxel interactions and produce voxel-wise features.
• Represent non-empty voxels as a sparse 4D tensor to exploit sparsity and enable efficient GPU processing.
• Apply 3D convolutional middle layers to aggregate context across voxels.
• Attach a Region Proposal Network (RPN) that operates on the dense voxel feature map to generate 3D detections.

Experimental results

Research questions

• RQ1 Can end-to-end learning directly on raw LiDAR points beat hand-crafted feature methods for 3D object detection?
• RQ2 Does voxel-based feature encoding enable effective 3D shape representation within sparse point clouds?
• RQ3 Is a LiDAR-only pipeline able to achieve state-of-the-art results on KITTI for cars, pedestrians, and cyclists?
• RQ4 What is the impact of end-to-end training versus hand-crafted baselines on detection performance in BEV and 3D spaces?

Key findings

• VoxelNet outperforms state-of-the-art LiDAR-based 3D detection methods on KITTI car detection across easy, moderate, and hard levels in 3D and BEV.
• In 3D detection, VoxelNet achieves higher AP than prior LiDAR+RGB method MV on all difficulty levels for cars.
• VoxelNet shows substantially better performance for pedestrians and cyclists in 3D detection compared to a hand-crafted baseline, highlighting the benefit of end-to-end learning for 3D shape representation.
• The KITTI test-set results indicate VoxelNet surpasses previous methods across bird’s-eye-view and 3D tasks, using only LiDAR data.
• An efficient implementation leverages sparse voxel representation and GPU-parallel stacked VFE layers to maintain tractable computation times.