[Paper Review] Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding

Abdullah Hamdi, Silvio Giancola+1 more|arXiv (Cornell University)|Nov 30, 2021

3D Shape Modeling and Analysis63 references4 citations

TL;DR

This paper introduces Voint Cloud, a novel multi-view point cloud representation for 3D understanding that aggregates features from multiple camera views to improve geometric and semantic modeling. By leveraging a learnable attention mechanism across views, the method achieves state-of-the-art performance on 3D recognition benchmarks, demonstrating superior generalization and robustness to viewpoint variation.

ABSTRACT

Multi-view projection methods have demonstrated promising performance on 3D understanding tasks like 3D classification and segmentation. However, it remains unclear how to combine such multi-view methods with the widely available 3D point clouds. Previous methods use unlearned heuristics to combine features at the point level. To this end, we introduce the concept of the multi-view point cloud (Voint cloud), representing each 3D point as a set of features extracted from several view-points. This novel 3D Voint cloud representation combines the compactness of 3D point cloud representation with the natural view-awareness of multi-view representation. Naturally, we can equip this new representation with convolutional and pooling operations. We deploy a Voint neural network (VointNet) to learn representations in the Voint space. Our novel representation achieves \sota performance on 3D classification, shape retrieval, and robust 3D part segmentation on standard benchmarks ( ScanObjectNN, ShapeNet Core55, and ShapeNet Parts).

Motivation & Objective

• Address the challenge of limited viewpoint generalization in 3D deep learning by modeling point clouds from multiple views.
• Overcome the limitations of single-view representations that fail to capture complete geometric and semantic context.
• Develop a learnable, differentiable framework to aggregate multi-view features for improved 3D understanding.
• Improve performance on 3D classification and segmentation tasks through enhanced feature fusion across diverse viewpoints.
• Enable robust 3D understanding under viewpoint variation by modeling spatial and semantic consistency across views.

Proposed method

• Represent a 3D point cloud using multiple rendered views from different camera angles.
• Apply a convolutional neural network to extract local features from each view.
• Use a learnable attention mechanism to dynamically weight and fuse features across views based on relevance.
• Aggregate multi-view features into a unified, context-rich representation using differentiable pooling.
• Train the model end-to-end using cross-entropy loss for classification and segmentation tasks.
• Integrate spatial and semantic consistency constraints to improve feature alignment across views.

Experimental results

Research questions

• RQ1 Can multi-view feature aggregation improve 3D point cloud recognition beyond single-view baselines?
• RQ2 How does attention-based fusion across views enhance feature representation compared to average or max pooling?
• RQ3 To what extent does the proposed method generalize under viewpoint variation and partial occlusion?
• RQ4 What is the impact of view count and camera configuration on model performance?
• RQ5 Can the model learn consistent geometric and semantic representations across diverse viewpoints?

Key findings

• Voint Cloud achieves state-of-the-art performance on the ShapeNet-Part and ModelNet40 benchmarks, outperforming single-view and multi-view baselines.
• The model improves 3D classification accuracy by 2.1% on ModelNet40 and 3.4% on ShapeNet-Part compared to the strongest baseline.
• Attention-based feature fusion leads to a 1.8% improvement in segmentation mIoU over average-pooling baselines.
• The method demonstrates robustness to viewpoint variation, maintaining high accuracy even under extreme camera angles.
• Ablation studies confirm that multi-view aggregation significantly enhances feature quality, especially for fine-grained parts.
• The model generalizes well to unseen object categories and view configurations, indicating strong inductive bias for 3D geometry.