[Paper Review] Deep Learning Based 3D Segmentation: A Survey

Yong He, Hongshan Yu+5 more|arXiv (Cornell University)|Mar 9, 2021

3D Shape Modeling and Analysis172 references44 citations

TL;DR

This survey comprehensively reviews deep learning methods for 3D segmentation across all data modalities (RGB-D, projected images, voxels, point clouds, meshes, and 3D videos), analyzing architectures, datasets, and future directions.

ABSTRACT

3D segmentation is a fundamental and challenging problem in computer vision with applications in autonomous driving and robotics. It has received significant attention from the computer vision, graphics and machine learning communities. Conventional methods for 3D segmentation, based on hand-crafted features and machine learning classifiers, lack generalization ability. Driven by their success in 2D computer vision, deep learning techniques have recently become the tool of choice for 3D segmentation tasks. This has led to an influx of many methods in the literature that have been evaluated on different benchmark datasets. Whereas survey papers on RGB-D and point cloud segmentation exist, there is a lack of a recent in-depth survey that covers all 3D data modalities and application domains. This paper fills the gap and comprehensively surveys the recent progress in deep learning-based 3D segmentation techniques. We cover over 220 works from the last six years, analyze their strengths and limitations, and discuss their competitive results on benchmark datasets. The survey provides a summary of the most commonly used pipelines and finally highlights promising research directions for the future.

Motivation & Objective

• Provide a unified overview of deep learning techniques for 3D segmentation across all 3D data modalities (RGB-D, projected images, voxels, point clouds, meshes, 3D videos).
• Analyze common building blocks, convolution kernels, and architectures used in 3D segmentation and discuss their strengths and limitations.
• Summarize benchmark datasets and evaluation metrics to enable fair comparisons across methods.
• Identify current challenges and propose promising directions for future research in 3D segmentation.

Proposed method

• Survey more than 180 representative works from the last five years focused on deep learning for 3D segmentation.
• Analyze and categorize methods by data representation and network architecture (RGB-D, projected images, voxel, point-based, 3D video, etc.).
• Discuss typical segmentation pipelines, including data encoding, fusion strategies, and post-processing steps (e.g., CRFs, GNNs, transformers).
• Provide benchmark comparisons on common 3D segmentation datasets and synthesize strengths/limitations of approaches.
• Highlight open challenges and future research directions in 3D segmentation.

Experimental results

Research questions

• RQ1 What are the relative advantages and limitations of 3D segmentation methods across different data representations (RGB-D, voxels, point clouds, meshes, 3D video)?
• RQ2 How do common architectural choices (e.g., MLP/PointNet-family, graph nets, transformers) impact performance and efficiency in 3D segmentation?
• RQ3 What datasets and evaluation metrics best reflect real-world performance, and how do methods compare on these benchmarks?
• RQ4 What are the key open challenges and promising directions for future research in deep learning-based 3D segmentation.

Key findings

• The survey covers over 180 works from the past five years and analyzes their strengths and limitations.
• It provides a comprehensive comparison of 3D segmentation methods across all major 3D data modalities.
• The paper discusses typical segmentation pipelines, including data representations, fusion strategies, and post-processing techniques.
• It identifies promising future research directions and practical considerations for benchmarking and evaluation.