[Paper Review] Deep Learning Advances on Different 3D Data Representations: A Survey.

Eman Ahmed, Alexandre Saint+6 more|arXiv (Cornell University)|Aug 4, 2018

3D Shape Modeling and AnalysisEngineering83 references56 citations

TL;DR

This survey provides a comprehensive analysis of deep learning applications across various 3D data representations, distinguishing between Euclidean and non-Euclidean formats. It examines how deep learning architectures are adapted to each representation, identifies key challenges, and evaluates performance trade-offs for tasks like segmentation, recognition, and correspondence.

ABSTRACT

3D data is a valuable asset the computer vision filed as it provides rich information about the full geometry of sensed objects and scenes. Recently, with the availability of both large 3D datasets and computational power, it is today possible to consider applying deep learning to learn specific tasks on 3D data such as segmentation, recognition and correspondence. Depending on the considered 3D data representation, different challenges may be foreseen in using existent deep learning architectures. In this work, we provide a comprehensive overview about various 3D data representations highlighting the difference between Euclidean and non-Euclidean ones. We also discuss how Deep Learning methods are applied on each representation, analyzing the challenges to overcome.

Motivation & Objective

• To systematically review and compare different 3D data representations used in computer vision.
• To identify the challenges arising when applying deep learning to Euclidean versus non-Euclidean 3D data structures.
• To analyze how existing deep learning models are adapted to various 3D representations for tasks such as segmentation, recognition, and correspondence.
• To highlight the performance differences and limitations of deep learning methods across distinct 3D data formats.

Proposed method

• Categorizing 3D data representations into Euclidean (e.g., voxel grids, point clouds) and non-Euclidean (e.g., meshes, graphs) formats.
• Surveying existing deep learning architectures tailored for each 3D representation type.
• Analyzing architectural adaptations required to handle geometric and topological differences in 3D data.
• Evaluating the effectiveness of convolutional, pooling, and attention mechanisms in different 3D data formats.
• Comparing training efficiency, inductive bias, and generalization performance across representations.
• Identifying gaps in current approaches related to data sparsity, irregularity, and structural complexity.

Experimental results

Research questions

• RQ1 How do Euclidean and non-Euclidean 3D data representations differ in their suitability for deep learning?
• RQ2 What are the key challenges in adapting deep learning models to irregular or sparse 3D data structures?
• RQ3 How do different 3D representations affect the performance of deep learning models in segmentation, recognition, and correspondence tasks?
• RQ4 What architectural modifications are necessary to effectively process non-Euclidean 3D data such as meshes or point clouds?
• RQ5 What are the trade-offs between representational fidelity, computational cost, and model generalization across 3D data formats?

Key findings

• Deep learning models show strong performance on structured 3D representations like voxel grids due to their regular grid structure.
• Point clouds and meshes require specialized architectures to handle sparsity and irregular topology, leading to increased complexity.
• Non-Euclidean representations such as meshes often preserve geometric fidelity better than voxel grids but pose challenges for standard convolutional operations.
• Graph-based and point-based networks demonstrate improved robustness to noise and varying sampling densities in 3D data.
• The choice of representation significantly impacts model efficiency, inductive bias, and generalization across different 3D vision tasks.
• Despite progress, challenges remain in achieving consistent performance across diverse 3D data formats, especially in handling occlusions and scale variations.