[Paper Review] SalsaNet: Fast Road and Vehicle Segmentation in LiDAR Point Clouds for Autonomous Driving

Eren Erdal Aksoy, Saimir Baci+1 more|arXiv (Cornell University)|Sep 18, 2019

Remote Sensing and LiDAR ApplicationsEnvironmental Science19 citations

TL;DR

SalsaNet is a fast, real-time deep encoder-decoder network for semantic segmentation of 3D LiDAR point clouds, projecting data into Bird's Eye View (BEV) for efficient 2D convolutional processing. It achieves state-of-the-art accuracy in road and vehicle segmentation while being projection-agnostic, outperforming prior methods in speed and IoU scores on the KITTI dataset using auto-labeled LiDAR data.

ABSTRACT

In this paper, we introduce a deep encoder-decoder network, named SalsaNet, for efficient semantic segmentation of 3D LiDAR point clouds. SalsaNet segments the road, i.e. drivable free-space, and vehicles in the scene by employing the Bird-Eye-View (BEV) image projection of the point cloud. To overcome the lack of annotated point cloud data, in particular for the road segments, we introduce an auto-labeling process which transfers automatically generated labels from the camera to LiDAR. We also explore the role of imagelike projection of LiDAR data in semantic segmentation by comparing BEV with spherical-front-view projection and show that SalsaNet is projection-agnostic. We perform quantitative and qualitative evaluations on the KITTI dataset, which demonstrate that the proposed SalsaNet outperforms other state-of-the-art semantic segmentation networks in terms of accuracy and computation time. Our code and data are publicly available at https://gitlab.com/aksoyeren/salsanet.git.

Motivation & Objective

• To develop a fast, accurate, and real-time semantic segmentation network for 3D LiDAR point clouds in autonomous driving.
• To address the scarcity of annotated LiDAR data—especially for road segments—by introducing a cross-sensor auto-labeling pipeline from camera to LiDAR.
• To evaluate and compare the impact of different LiDAR point cloud projections (BEV vs. SFV) on semantic segmentation performance.
• To design a model that is robust and performs well across different projection formats, achieving high accuracy and inference speed.
• To release code and a large-scale auto-labeled LiDAR dataset to support future research.

Proposed method

• SalsaNet employs an encoder-decoder architecture with residual blocks in the encoder and skip connections to fuse features from early and late layers in the decoder.
• The input to SalsaNet is a BEV-projected LiDAR point cloud, where each channel encodes a statistical feature (e.g., depth, intensity, range, occupancy mask).
• The network uses a multi-channel input representation to preserve spatial and intensity information, enabling effective feature learning in 2D grid space.
• An auto-labeling pipeline transfers semantic labels from camera images (road and vehicles) to LiDAR point clouds using sensor calibration, generating ~11,000 additional annotated samples.
• The model is trained and evaluated on the KITTI dataset, with ablation studies on loss weighting and channel contributions.
• Performance is compared across BEV and spherical-front-view (SFV) projections to assess projection robustness.

Experimental results

Research questions

• RQ1 How does the choice of LiDAR point cloud projection (BEV vs. SFV) affect semantic segmentation accuracy and speed?
• RQ2 Can a deep learning model achieve state-of-the-art performance in LiDAR-based road and vehicle segmentation while maintaining real-time inference?
• RQ3 To what extent can auto-labeling from camera data mitigate the scarcity of annotated LiDAR data?
• RQ4 Is the proposed network architecture robust to different input projections, making it projection-agnostic?
• RQ5 What is the impact of individual input channels (e.g., x, y, z, intensity, range, mask) on segmentation performance?

Key findings

• SalsaNet achieves a mean Intersection over Union (IoU) of 79.71% on the KITTI dataset using BEV projection, outperforming prior state-of-the-art methods.
• The network achieves 160 Hz inference speed on a Tesla V100 GPU, significantly faster than the 10 Hz sampling rate of typical LiDAR sensors.
• SalsaNet demonstrates projection-agnostic performance, maintaining high accuracy in both BEV and SFV projections, unlike other models that struggle with BEV.
• The SFV projection introduces redundant information that degrades performance; for example, adding the mask channel improves IoU by 4.5%, while x and y coordinates reduce accuracy by 0.04% and 0.34% respectively.
• Ablation studies show that loss weighting significantly impacts IoU, with unweighted loss reducing average IoU by 2.3%.
• The auto-labeling pipeline successfully generates ~11,000 additional annotated LiDAR samples, enabling robust training without manual annotation.