[Paper Review] MASS: Multi-Attentional Semantic Segmentation of LiDAR Data for Dense Top-View Understanding

Kunyu Peng, Juncong Fei+7 more|arXiv (Cornell University)|Jul 1, 2021

Advanced Neural Network Applications4 citations

TL;DR

MASS proposes a multi-attentional deep learning framework for dense top-view semantic segmentation of LiDAR data, leveraging pillar features and three attention mechanisms—keypoint-driven graph, LSTM-based spatial, and pillar-based attention—to achieve state-of-the-art performance. It achieves 58.80% mIoU on SemanticKITTI and 30.4% on nuScenes-LidarSeg, demonstrating superior generalization and near real-time inference for automated driving applications.

ABSTRACT

At the heart of all automated driving systems is the ability to sense the surroundings, e.g., through semantic segmentation of LiDAR sequences, which experienced a remarkable progress due to the release of large datasets such as SemanticKITTI and nuScenes-LidarSeg. While most previous works focus on sparse segmentation of the LiDAR input, dense output masks provide self-driving cars with almost complete environment information. In this paper, we introduce MASS - a Multi-Attentional Semantic Segmentation model specifically built for dense top-view understanding of the driving scenes. Our framework operates on pillar- and occupancy features and comprises three attention-based building blocks: (1) a keypoint-driven graph attention, (2) an LSTM-based attention computed from a vector embedding of the spatial input, and (3) a pillar-based attention, resulting in a dense 360-degree segmentation mask. With extensive experiments on both, SemanticKITTI and nuScenes-LidarSeg, we quantitatively demonstrate the effectiveness of our model, outperforming the state of the art by 19.0% on SemanticKITTI and reaching 30.4% in mIoU on nuScenes-LidarSeg, where MASS is the first work addressing the dense segmentation task. Furthermore, our multi-attention model is shown to be very effective for 3D object detection validated on the KITTI-3D dataset, showcasing its high generalizability to other tasks related to 3D vision.

Motivation & Objective

• To address the challenge of dense top-view semantic segmentation from sparse LiDAR point clouds, which enables richer environmental understanding than sparse point-wise segmentation.
• To overcome limitations of 2D CNNs on irregular, sparse 3D point clouds by designing a pillar-based, end-to-end framework with attention mechanisms.
• To improve generalization and robustness in real-world driving scenarios through multi-scale data augmentation and attention-based feature aggregation.
• To demonstrate the model's transferability to related 3D vision tasks, such as 3D object detection, via ablation and cross-dataset evaluation.

Proposed method

• The framework uses a pillar feature network (PFN) to convert sparse 3D LiDAR point clouds into dense 2D pillar features, enabling efficient 2D CNN processing.
• A modified UNet (M-UNet) backbone processes the pillar features to produce dense top-view segmentation masks.
• Three attention mechanisms are integrated: (1) keypoint-driven graph attention for local context modeling, (2) LSTM-based attention from spatial vector embeddings for sequential dependency learning, and (3) pillar-based attention for global feature refinement.
• The model is trained end-to-end using cross-entropy loss on grid-cell-level labels, with supervision constrained to the visible region via an occupancy map.
• Data augmentation techniques—random flip, rotation, and scale—are applied to enhance robustness, while translation is excluded due to performance degradation.
• Inference is optimized for real-time deployment, achieving 74ms per inference on a GTX 2080Ti, with MA adding only ~16ms overhead.

Experimental results

Research questions

• RQ1 Can a multi-attention mechanism improve dense top-view semantic segmentation performance on sparse LiDAR data compared to existing sparse or 2D segmentation methods?
• RQ2 How effective are the three proposed attention modules—keypoint-driven graph, LSTM-based spatial, and pillar-based attention—in capturing local and global context for 3D scene understanding?
• RQ3 Does the proposed MASS framework generalize well to other 3D vision tasks, such as 3D object detection, beyond semantic segmentation?
• RQ4 To what extent do data augmentation strategies like random flip, rotation, and scale improve model generalization and robustness in real-world driving scenarios?

Key findings

• MASS achieves 58.80% mIoU on the SemanticKITTI dataset, outperforming the state of the art by 19.0% in mIoU.
• On the nuScenes-LidarSeg benchmark, MASS sets a new SOTA with 30.4% mIoU, being the first method to address dense top-view segmentation on this dataset.
• The ablation study confirms that random flip and rotation improve mIoU by 4.6%, while scale adds 0.6%, demonstrating significant gains from augmentation.
• The model achieves near real-time inference at 74ms per frame on a GTX 2080Ti, making it suitable for real-time automated driving applications.
• The multi-attention mechanism enhances feature representation, as shown by consistent improvements across ablation studies and downstream 3D object detection tasks.
• The framework generalizes well to 3D object detection, as validated on the KITTI-3D dataset, indicating strong transferability across 3D vision tasks.