[Paper Review] End-to-End 3D-PointCloud Semantic Segmentation for Autonomous Driving

Mohammed Abdou, Mahmoud Elkhateeb+2 more|arXiv (Cornell University)|Jun 26, 2019

3D Shape Modeling and Analysis22 references4 citations

TL;DR

This paper proposes Weighted Self-Incremental Transfer Learning to address class imbalance in 3D point cloud semantic segmentation for autonomous driving. By combining class-weighted loss with incremental training on non-dominant classes first, the method significantly improves segmentation of rare classes like pedestrians and cyclists, achieving a 39.2% overall mean IoU improvement on the KITTI benchmark compared to baseline PointNet++.

ABSTRACT

3D semantic scene labeling is a fundamental task for Autonomous Driving. Recent work shows the capability of Deep Neural Networks in labeling 3D point sets provided by sensors like LiDAR, and Radar. Imbalanced distribution of classes in the dataset is one of the challenges that face 3D semantic scene labeling task. This leads to misclassifying for the non-dominant classes which suffer from two main problems: a) rare appearance in the dataset, and b) few sensor points reflected from one object of these classes. This paper proposes a Weighted Self-Incremental Transfer Learning as a generalized methodology that solves the imbalanced training dataset problems. It re-weights the components of the loss function computed from individual classes based on their frequencies in the training dataset, and applies Self-Incremental Transfer Learning by running the Neural Network model on non-dominant classes first, then dominant classes one-by-one are added. The experimental results introduce a new 3D point cloud semantic segmentation benchmark for KITTI dataset.

Motivation & Objective

• Address the challenge of class imbalance in 3D point cloud datasets, particularly in autonomous driving applications.
• Overcome the limitations of existing methods that fail to accurately segment rare, fine-grained classes like pedestrians and cyclists.
• Develop a generalized, end-to-end deep learning methodology that enhances model generalization and prevents catastrophic forgetting during incremental learning.
• Establish a new benchmark for 3D semantic segmentation on the KITTI dataset using the proposed methodology.

Proposed method

• Apply a weighted cross-entropy loss function that assigns higher penalties to underrepresented classes based on their class frequency and point density in the dataset.
• Implement a self-incremental transfer learning strategy that trains first on non-dominant, fine-grained classes (e.g., pedestrians, cyclists) before gradually adding dominant classes (e.g., cars, trucks).
• Use a pre-trained model from earlier phases as the initialization for subsequent training phases, ensuring knowledge retention without forgetting.
• Calculate class weights based on two factors: (1) the number of dataset instances per class, and (2) the number of LiDAR-reflected points per class.
• Train the PointNet++ backbone using the combined weighted loss and incremental class addition procedure, avoiding reliance on external pre-trained models.
• Evaluate performance using mean Intersection over Union (mIoU) across all classes, with ablation studies comparing weighted loss only, self-incremental only, and the full method.

Experimental results

Research questions

• RQ1 Can a weighted loss function alone effectively mitigate class imbalance in 3D semantic segmentation of sparse LiDAR point clouds?
• RQ2 Does self-incremental transfer learning alone prevent catastrophic forgetting while improving segmentation of rare classes in imbalanced datasets?
• RQ3 Can combining weighted loss with incremental training on non-dominant classes significantly improve mIoU for fine-grained classes like pedestrians and cyclists?
• RQ4 Does the proposed method generalize across different class categories and maintain robustness on real-world autonomous driving datasets like KITTI?

Key findings

• The baseline PointNet++ model fails to classify pedestrians and cyclists, achieving near-zero mIoU for these classes on the KITTI dataset.
• Using only a weighted loss function does not resolve the imbalance issue, as the model still misclassifies rare classes due to insufficient representation learning.
• Self-incremental transfer learning alone leads to catastrophic forgetting, where the model loses accuracy on previously learned classes (e.g., pedestrians and cyclists) when new dominant classes like cars are added.
• The proposed Weighted Self-Incremental Transfer Learning method achieves a 6.71% mIoU improvement for the car class and 11.17% for the van class compared to the baseline.
• Pedestrian and cyclist classes, previously unclassified, achieve mIoU scores of 15.33% and 22.96%, respectively, under the proposed method.
• The overall mean IoU increases by 39.2% compared to the baseline, demonstrating the method’s effectiveness in handling class imbalance and enabling stable, generalizable 3D semantic segmentation.