[Paper Review] Review: deep learning on 3D point clouds

Saifullahi Aminu Bello, Shangshu Yu+1 more|arXiv (Cornell University)|Jan 17, 2020

3D Shape Modeling and Analysis102 references4 citations

TL;DR

This paper reviews state-of-the-art deep learning methods for 3D point cloud processing, focusing on direct learning from unstructured point clouds to overcome limitations of grid-based preprocessing. It surveys architectures like PointNet and its variants that capture local structures via sampling, grouping, and MLP-based feature learning, achieving strong performance on classification, segmentation, and detection benchmarks such as ModelNet40 and KITTI.

ABSTRACT

Point cloud is point sets defined in 3D metric space. Point cloud has become one of the most significant data format for 3D representation. Its gaining increased popularity as a result of increased availability of acquisition devices, such as LiDAR, as well as increased application in areas such as robotics, autonomous driving, augmented and virtual reality. Deep learning is now the most powerful tool for data processing in computer vision, becoming the most preferred technique for tasks such as classification, segmentation, and detection. While deep learning techniques are mainly applied to data with a structured grid, point cloud, on the other hand, is unstructured. The unstructuredness of point clouds makes use of deep learning for its processing directly very challenging. Earlier approaches overcome this challenge by preprocessing the point cloud into a structured grid format at the cost of increased computational cost or lost of depth information. Recently, however, many state-of-the-arts deep learning techniques that directly operate on point cloud are being developed. This paper contains a survey of the recent state-of-the-art deep learning techniques that mainly focused on point cloud data. We first briefly discussed the major challenges faced when using deep learning directly on point cloud, we also briefly discussed earlier approaches which overcome the challenges by preprocessing the point cloud into a structured grid. We then give the review of the various state-of-the-art deep learning approaches that directly process point cloud in its unstructured form. We introduced the popular 3D point cloud benchmark datasets. And we also further discussed the application of deep learning in popular 3D vision tasks including classification, segmentation and detection.

Motivation & Objective

• To address the challenges of applying deep learning directly to unstructured, irregular, and unordered 3D point clouds.
• To survey early approaches that convert point clouds into structured grids, highlighting their limitations in computational cost and information loss.
• To provide a detailed review of recent state-of-the-art deep learning methods that process point clouds directly in their native form.
• To present key 3D point cloud benchmark datasets and evaluate performance across major 3D vision tasks.
• To identify open research directions, such as instance segmentation and large-scale scene processing, in 3D point cloud learning.

Proposed method

• Categorize challenges in deep learning on point clouds into irregularity, unstructured nature, and unordered point sets.
• Review preprocessing-based methods that convert point clouds into voxel grids or bird’s-eye views to enable CNN usage.
• Analyze direct point cloud processing methods, particularly PointNet and its extensions, which use permutation-invariant operations like max pooling.
• Explain the core pipeline of local feature extraction: sampling to select centroids, grouping neighboring points via K-NN, and applying MLPs or PointNet-like modules to learn local representations.
• Highlight architectural innovations that model inter-point relationships within local patches to improve feature discriminability.
• Compare methods based on modality (LiDAR, RGB-D, multi-modal), speed, and performance on standard benchmarks like ModelNet40 and KITTI.

Experimental results

Research questions

• RQ1 What are the main challenges in applying deep learning directly to unstructured 3D point clouds?
• RQ2 How do preprocessing-based methods overcome these challenges, and what are their key limitations?
• RQ3 What are the core architectural principles enabling direct deep learning on point clouds, and how do they preserve local and global structure?
• RQ4 How do recent methods improve upon PointNet in capturing local geometric relationships?
• RQ5 What are the current performance limits and open challenges in 3D point cloud tasks such as instance segmentation and large-scale scene understanding?

Key findings

• PointNet and its variants achieve strong performance on 3D point cloud classification and segmentation, with PointNet++ improving local feature learning through hierarchical sampling and grouping.
• On the ModelNet40 benchmark, PointNet achieves a classification accuracy of 89.2% using only XYZ coordinates, demonstrating the effectiveness of permutation-invariant learning.
• For 3D object detection on the KITTI dataset, PointPillars achieves a mAP of 59.2% on the 3D detection benchmark and 56.0% on the Bird’s Eye View benchmark, with inference speed of 62 Hz.
• Methods like PointRCNN and VoxelNet achieve mAP scores above 80% on the KITTI 3D detection benchmark, indicating high accuracy in object detection from point clouds.
• Despite progress, instance segmentation and direct 3D detection from point clouds remain underexplored, with only a few works like [120, 121] addressing large-scale scene processing.
• The use of multi-modal inputs (LiDAR and image) improves detection performance, with methods like AVOD-FPN achieving mAP of 55.62% on the KITTI 3D benchmark.