[Paper Review] Vehicle Detection from 3D Lidar Using Fully Convolutional Network

Bo Li, Tianlei Zhang+1 more|arXiv (Cornell University)|Aug 29, 2016

Advanced Optical Sensing TechnologiesPhysics and Astronomy21 references275 citations

TL;DR

The paper presents an end-to-end 2D FCN that processes Velodyne 64E range scans projected to a 2D map to predict objectness and 3D bounding boxes for vehicles, achieving state-of-the-art results on KITTI range-scan based detection.

ABSTRACT

Convolutional network techniques have recently achieved great success in vision based detection tasks. This paper introduces the recent development of our research on transplanting the fully convolutional network technique to the detection tasks on 3D range scan data. Specifically, the scenario is set as the vehicle detection task from the range data of Velodyne 64E lidar. We proposes to present the data in a 2D point map and use a single 2D end-to-end fully convolutional network to predict the objectness confidence and the bounding boxes simultaneously. By carefully design the bounding box encoding, it is able to predict full 3D bounding boxes even using a 2D convolutional network. Experiments on the KITTI dataset shows the state-of-the-art performance of the proposed method.

Motivation & Objective

• Motivate applying fully convolutional networks to 3D LiDAR range scans for vehicle detection.
• Project 3D LiDAR points to a 2D point map to enable end-to-end 2D FCN processing.
• Simultaneously predict objectness and full 3D bounding boxes within a unified framework.
• Design a rotation-invariant bounding box encoding to handle viewpoint changes.
• Achieve competitive or state-of-the-art performance on KITTI using end-to-end learning.

Proposed method

• Convert Velodyne 64E LiDAR points into a 2D point map with channels (d, z).
• Use a 2D fully convolutional network with a shared trunk and two heads: objectness classification and 24D bounding box regression.
• Encode 3D bounding boxes by transforming corner coordinates with a rotation R and reporting 8 corners per object.
• Concatenate feature maps from multiple layers to improve small object and edge prediction.
• Apply data augmentation via geometry-preserving 3D transforms and multi-task loss balancing for objectness and box regression.
• Train with weighted losses to balance foreground/background and varying object sizes/distances.

Experimental results

Research questions

• RQ1 Can a fully convolutional network operate directly on 2D projections of 3D LiDAR range scans to detect vehicles?
• RQ2 How can 3D bounding boxes be encoded so that a 2D convolutional backbone can predict full 3D boxes?
• RQ3 Does joint objectness and 3D box regression improve detection in LiDAR data compared to traditional proposal-based methods?
• RQ4 What dataset and evaluation protocol (KITTI) can demonstrate state-of-the-art performance for LiDAR-based vehicle detection?
• RQ5 How do data augmentation and loss balancing affect training on sparse LiDAR point clouds?

Key findings

• The proposed FCN achieves high detection performance on KITTI range-scan data, with state-of-the-art offline world-space AP/AOS in Easy settings.
• Offline world-space AP: Easy 77.3%, Moderate 72.4%, Hard 69.4%; Offline world-space AOS: Easy 77.2%, Moderate 72.3%, Hard 69.4%.
• Image-space AP/AOS are slightly lower than world-space, reflecting differences between 2D projection overlap and 3D localization.
• The method can predict complete 3D bounding boxes even for vehicles partly visible, aiding tracking and planning.
• Compared to prior range-scan methods, the approach shows improved AP in Easy and competitive AP in Moderate/Hard tasks; it also achieves superior orientation estimation (AOS).
• The network benefits from multi-layer feature concatenation to improve small object and edge predictions.