[Paper Review] Semantic Segmentation of 3D LiDAR Data in Dynamic Scene Using Semi-supervised Learning

Jilin Mei, Biao Gao+4 more|arXiv (Cornell University)|Sep 3, 2018

Human Pose and Action Recognition3 citations

TL;DR

This paper proposes a semi-supervised learning framework for 3D LiDAR semantic segmentation in dynamic scenes, leveraging inter-frame data association to generate inexpensive pairwise constraints and a novel loss function combining annotations and constraints. The method achieves over 10% improvement in segmentation accuracy using only a few human annotations and a large volume of constraint data, significantly enhancing model adaptability to new scenes.

ABSTRACT

This work studies the semantic segmentation of 3D LiDAR data in dynamic scenes for autonomous driving applications. A system of semantic segmentation using 3D LiDAR data, including range image segmentation, sample generation, inter-frame data association, track-level annotation and semi-supervised learning, is developed. To reduce the considerable requirement of fine annotations, a CNN-based classifier is trained by considering both supervised samples with manually labeled object classes and pairwise constraints, where a data sample is composed of a segment as the foreground and neighborhood points as the background. A special loss function is designed to account for both annotations and constraints, where the constraint data are encouraged to be assigned to the same semantic class. A dataset containing 1838 frames of LiDAR data, 39934 pairwise constraints and 57927 human annotations is developed. The performance of the method is examined extensively. Qualitative and quantitative experiments show that the combination of a few annotations and large amount of constraint data significantly enhances the effectiveness and scene adaptability, resulting in greater than 10% improvement

Motivation & Objective

• Address the high cost of pixel-wise annotation in 3D LiDAR semantic segmentation for autonomous driving.
• Improve model generalization and adaptability to new, unseen scenes with minimal retraining.
• Leverage inter-frame data association from moving objects to generate low-cost, abundant pairwise constraints.
• Design a joint loss function that incorporates both sparse human annotations and dense pairwise constraints.
• Develop a new dataset with 1838 frames, 39,934 constraints, and 57,927 human annotations to evaluate the method.

Proposed method

• Construct samples as a foreground segment (e.g., a moving object) paired with surrounding background points.
• Use a CNN-based classifier trained on both manually labeled data and pairwise constraints indicating same/different class membership.
• Design a custom loss function that minimizes classification disagreement for constrained pairs while respecting ground-truth labels.
• Apply ego-motion compensation to align consecutive LiDAR frames for inter-frame data association.
• Use a tracking algorithm to associate the same object across frames, generating positive pairwise constraints.
• Fine-tune the model using a small set of human-confirmed anchor samples after initial semi-supervised training.

Experimental results

Research questions

• RQ1 Can pairwise constraints derived from tracked objects in consecutive LiDAR frames effectively improve semantic segmentation without additional human annotations?
• RQ2 How does combining a small number of human annotations with a large number of automatically generated constraints affect model performance and generalization?
• RQ3 To what extent can a semi-supervised model trained on one scene adapt to a new, unseen scene with minimal retraining?
• RQ4 Does unsupervised learning using only constraints fail, and if so, why?
• RQ5 Can fine-tuning with a few human-confirmed samples significantly improve model accuracy and adaptability in new environments?

Key findings

• The addition of 100 pairwise constraints improved the average F-score by 9% compared to a supervised baseline trained on the same data.
• The semi-supervised method with constraints achieved an average F-score increase of over 10% compared to fully supervised baselines.
• Unsupervised training using only constraints failed because the loss function only penalized misclassification of constraint pairs, not overall class distribution.
• Fine-tuning with 20 human-confirmed samples per category and 100 for the unknown class increased the average F-score by 13% compared to the constraint-only model.
• The fine-tuned model outperformed the supervised baseline on most categories, especially for challenging classes like cyclist and trunk, despite lower performance on the training data.
• Qualitative results showed that the fine-tuned model successfully classified previously misclassified objects such as cyclists and cars in new scenes.