[Paper Review] Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey and Experimental Study

Biao Gao, Yancheng Pan+3 more|arXiv (Cornell University)|Jun 8, 2020

Advanced Neural Network Applications220 references4 citations

TL;DR

This paper investigates the data hunger problem in 3D LiDAR semantic segmentation, analyzing dataset size, diversity, and domain gaps across S3DIS, Semantic3D, and SemanticKITTI. It finds that long-tailed distributions, limited scene diversity, and significant domain gaps hinder model generalization, and calls for improved data metrics, domain-aware training, and standardized category definitions to address persistent data scarcity challenges in autonomous systems.

ABSTRACT

3D semantic segmentation is a fundamental task for robotic and autonomous driving applications. Recent works have been focused on using deep learning techniques, whereas developing fine-annotated 3D LiDAR datasets is extremely labor intensive and requires professional skills. The performance limitation caused by insufficient datasets is called data hunger problem. This research provides a comprehensive survey and experimental study on the question: are we hungry for 3D LiDAR data for semantic segmentation? The studies are conducted at three levels. First, a broad review to the main 3D LiDAR datasets is conducted, followed by a statistical analysis on three representative datasets to gain an in-depth view on the datasets' size and diversity, which are the critical factors in learning deep models. Second, a systematic review to the state-of-the-art 3D semantic segmentation is conducted, followed by experiments and cross examinations of three representative deep learning methods to find out how the size and diversity of the datasets affect deep models' performance. Finally, a systematic survey to the existing efforts to solve the data hunger problem is conducted on both methodological and dataset's viewpoints, followed by an insightful discussion of remaining problems and open questions To the best of our knowledge, this is the first work to analyze the data hunger problem for 3D semantic segmentation using deep learning techniques that are addressed in the literature review, statistical analysis, and cross-dataset and cross-algorithm experiments. We share findings and discussions, which may lead to potential topics in future works.

Motivation & Objective

• To investigate whether deep learning models for 3D LiDAR semantic segmentation suffer from data hunger due to insufficient or imbalanced training data.
• To analyze the size, diversity, and distribution characteristics of major 3D LiDAR datasets (S3DIS, Semantic3D, SemanticKITTI) through statistical analysis.
• To evaluate how dataset size and diversity impact the performance of state-of-the-art deep learning models in 3D semantic segmentation.
• To survey existing methodological and dataset-level solutions to the data hunger problem and identify open challenges for future research.
• To advocate for standardized category definitions, domain gap metrics, and uncertainty-aware models to improve robustness in real-world deployment.

Proposed method

• Conducted a comprehensive survey of 3D LiDAR datasets, focusing on data size, class distribution, and scene diversity.
• Performed statistical analysis on three representative datasets (S3DIS, Semantic3D, SemanticKITTI) to quantify long-tailed class distributions and spatial imbalances.
• Selected three state-of-the-art deep learning models (e.g., PointNet++, PointConv, Point-NeXt) for cross-dataset training and testing to assess generalization under data scarcity.
• Executed cross-dataset experiments to evaluate model performance when trained on one dataset and tested on another, revealing domain gap effects.
• Surveyed methodological approaches such as data augmentation, self-supervised learning, and domain adaptation, as well as dataset curation strategies.
• Proposed the need for quantitative domain gap metrics and standardized category definitions to enable reliable model evaluation and dataset sharing.

Experimental results

Research questions

• RQ1 To what extent do current 3D LiDAR datasets exhibit long-tailed class distributions and spatial imbalances that hinder model generalization?
• RQ2 How does the diversity of scenes within and across datasets affect the performance and robustness of deep learning models in 3D semantic segmentation?
• RQ3 What is the impact of domain gaps between datasets on model generalization, and does mixing datasets improve performance?
• RQ4 What methodological and dataset-level strategies are currently available to mitigate data hunger, and what are their limitations?
• RQ5 What open problems remain in creating standardized, scalable, and diverse 3D LiDAR datasets for robust semantic segmentation in real-world applications?

Key findings

• 3D LiDAR datasets exhibit severe long-tailed class distributions, with a large proportion of points belonging to dominant classes like 'road' near the sensor's viewpoint.
• Despite their popularity, S3DIS, Semantic3D, and SemanticKITTI show insufficient inner-dataset diversity and large cross-dataset differences, limiting model generalization.
• Mixing multiple datasets for training does not necessarily improve model accuracy due to significant domain gaps between them.
• Testing models on datasets with large domain gaps leads to substantial performance degradation, reducing the reliability of benchmark evaluations.
• The sensory and semantic gaps—such as occlusion, sparse sampling, and functional similarity across object types—pose fundamental challenges for consistent annotation and generalization.
• There is a critical need for standardized category definitions and quantitative domain gap metrics to enable effective dataset sharing and model evaluation across domains.