[Paper Review] Multiview Based 3D Scene Understanding On Partial Point Sets

Ye Zhu, Sven Ewan Shepstone+4 more|arXiv (Cornell University)|Nov 30, 2018

Advanced Vision and Imaging17 references4 citations

TL;DR

This paper proposes a 3D multiview representation that enhances deep learning models for 3D scene understanding on both complete and partial point clouds. By augmenting training data with multiview-based partial point sets derived from 360° scans, the method improves semantic segmentation accuracy by 31.9% on partial sets and 4.3% on complete sets, while also generalizing to 3D part segmentation.

ABSTRACT

Deep learning within the context of point clouds has gained much research interest in recent years mostly due to the promising results that have been achieved on a number of challenging benchmarks, such as 3D shape recognition and scene semantic segmentation. In many realistic settings however, snapshots of the environment are often taken from a single view, which only contains a partial set of the scene due to the field of view restriction of commodity cameras. 3D scene semantic understanding on partial point clouds is considered as a challenging task. In this work, we propose a processing approach for 3D point cloud data based on a multiview representation of the existing 360° point clouds. By fusing the original 360° point clouds and their corresponding 3D multiview representations as input data, a neural network is able to recognize partial point sets while improving the general performance on complete point sets, resulting in an overall increase of 31.9% and 4.3% in segmentation accuracy for partial and complete scene semantic understanding, respectively. This method can also be applied in a wider 3D recognition context such as 3D part segmentation.

Motivation & Objective

• Address the challenge of 3D scene semantic understanding on partial point clouds, which are common in real-world applications due to field-of-view limitations of cameras.
• Overcome the limitation of existing deep learning models trained only on complete 360° point clouds, which struggle to generalize to partial, real-world scans.
• Develop a data representation strategy that enhances model robustness to partial observations without relying solely on data augmentation.
• Demonstrate the effectiveness of multiview-based partial point sets as a general-purpose preprocessing technique for 3D recognition tasks beyond scene segmentation.

Proposed method

• Generate multiview-based partial point sets by projecting a 360° point cloud from multiple viewpoints, simulating real-world camera snapshots with field-of-view constraints.
• Maintain the 3D geometric structure of the original point clouds rather than converting to 2D images, preserving spatial relationships for deep learning.
• Train a deep neural network using a combined dataset of original 360° point clouds and their multiview-derived partial sets to improve generalization.
• Use a theoretical framework to show that if the critical point set of the full scene can be reconstructed from partial views, the full data can be replaced by multiview sets.
• Apply the method to both scene semantic segmentation and 3D part segmentation tasks, validating its generalization across tasks.
• Optimize the number of multiview partial sets to balance performance on complete and partial point cloud recognition, identifying a performance trade-off.

Experimental results

Research questions

• RQ1 Can a deep neural network trained on 360° point clouds generalize effectively to partial point sets captured from real-world cameras with limited field of view?
• RQ2 To what extent does incorporating multiview-based partial point sets improve the performance of 3D semantic segmentation on both complete and incomplete data?
• RQ3 Is the proposed multiview representation effective even when the number of training samples is not increased, indicating it is not merely a data augmentation trick?
• RQ4 Can the multiview representation be generalized beyond scene-level semantic segmentation to fine-grained tasks like 3D part segmentation?
• RQ5 What trade-off exists between the number of multiview partial sets and the model’s ability to recognize both complete and partial point clouds?

Key findings

• The proposed method increases mean Intersection over Union (mIoU) by 31.9% on partial point cloud semantic segmentation compared to the baseline.
• The method also improves performance on complete 360° point clouds by 4.3% in mIoU, demonstrating dual benefit for both partial and complete data.
• Even when the original 360° point clouds are completely removed from training and replaced with 300 multiview partial sets per scene, the model achieves comparable performance on complete data.
• The method generalizes effectively to 3D part segmentation, achieving consistent mIoU improvements across 15 out of 16 object categories on the ShapeNet part dataset.
• A performance trade-off is observed: increasing the number of multiview partial sets initially improves recognition on both complete and partial data, but beyond a point, performance on complete sets begins to decline.
• Theoretical analysis confirms that if the critical point set of the full scene can be reconstructed from partial views, the full data can be replaced by multiview sets without performance loss.