[Paper Review] SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud

BoRui Wu, Alvin Wan+2 more|arXiv (Cornell University)|Oct 19, 2017

Advanced Neural Network ApplicationsComputer Science16 references56 citations

TL;DR

SqueezeSeg presents an end-to-end CNN pipeline on spherical-projected LiDAR data with a recurrent CRF refinement for real-time road-object segmentation, achieving high accuracy and fast frame rates on KITTI, with added benefits from synthesized GTA-V data.

ABSTRACT

In this paper, we address semantic segmentation of road-objects from 3D LiDAR point clouds. In particular, we wish to detect and categorize instances of interest, such as cars, pedestrians and cyclists. We formulate this problem as a point- wise classification problem, and propose an end-to-end pipeline called SqueezeSeg based on convolutional neural networks (CNN): the CNN takes a transformed LiDAR point cloud as input and directly outputs a point-wise label map, which is then refined by a conditional random field (CRF) implemented as a recurrent layer. Instance-level labels are then obtained by conventional clustering algorithms. Our CNN model is trained on LiDAR point clouds from the KITTI dataset, and our point-wise segmentation labels are derived from 3D bounding boxes from KITTI. To obtain extra training data, we built a LiDAR simulator into Grand Theft Auto V (GTA-V), a popular video game, to synthesize large amounts of realistic training data. Our experiments show that SqueezeSeg achieves high accuracy with astonishingly fast and stable runtime (8.7 ms per frame), highly desirable for autonomous driving applications. Furthermore, additionally training on synthesized data boosts validation accuracy on real-world data. Our source code and synthesized data will be open-sourced.

Motivation & Objective

• Develop an end-to-end CNN-based pipeline for point-wise semantic segmentation of road objects from 3D LiDAR point clouds.
• Reduce memory and computation using a lightweight architecture inspired by SqueezeNet.
• Enhance segmentation accuracy with a CRF as a recurrent layer to refine label maps.
• Leverage synthetic data from GTA-V to augment real-world KITTI data and improve performance.

Proposed method

• Project 3D LiDAR points onto a dense 2D spherical grid to create an image-like input for CNNs.
• Adapt SqueezeNet-inspired architecture with fire modules and fireDeconvs to achieve real-time, low-parameter segmentation.
• Produce per-point label probabilities with a final softmax layer to generate a point-wise label map.
• Refine the probabilistic label map using a mean-field CRF implemented as an RNN, allowing end-to-end training.
• Optionally apply clustering (e.g., DBSCAN) on labeled points for instance-level segmentation.
• Train and evaluate on KITTI, and augment training with GTA-V synthesized LiDAR data to improve accuracy.

Experimental results

Research questions

• RQ1 Can a lightweight CNN operating on sphere-projected LiDAR data achieve accurate, real-time semantic segmentation of road objects?
• RQ2 Does integrating a recurrent CRF improve boundary precision and overall IoU for LiDAR-based segmentation?
• RQ3 What is the impact of synthetic data from GTA-V on KITTI-based performance?
• RQ4 Is end-to-end training feasible for the combined CNN-CRF pipeline for 3D LiDAR segmentation?

Key findings

• SqueezeSeg with CRF achieves higher IoU for the car class compared to without CRF (64.6 vs 60.9 on class-level, car with CRF).
• CRF improves precision, boosting instance-level car IoU and P/R metrics (car with CRF: IoU 59.5; without CRF: IoU 56.7).
• Car class shows high recall (>90%) in both with/without CRF configurations, indicating low false negatives.
• Pedestrian and cyclist performance are lower due to fewer instances and smaller object sizes; CRF provides mixed gains.
• Real-time runtime: SqueezeSeg without CRF 8.7 ms/frame; with CRF 13.5 ms/frame on TITAN X; DBSCAN clustering for instances adds additional cost (27.3 ms on average).
• Training with GTA-synthesized data plus KITTI yields further IoU gains for car segmentation compared to KITTI alone.