[Paper Review] Multi Point-Voxel Convolution (MPVConv) for Deep Learning on Point Clouds

Wei Zhou, Xin Cao+4 more|arXiv (Cornell University)|Jul 28, 2021

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper proposes Multi Point-Voxel Convolution (MPVConv), a novel 3D CNN module that combines point-based and voxel-based features using 3D CNNs and MLPs to enhance neighborhood aggregation in point features and independence in voxel features. MPVConv boosts PointNet accuracy by up to 36% and achieves up to 34× speedups over voxel-based models, outperforming state-of-the-art models with lower latency.

ABSTRACT

The existing 3D deep learning methods adopt either individual point-based features or local-neighboring voxel-based features, and demonstrate great potential for processing 3D data. However, the point based models are inefficient due to the unordered nature of point clouds and the voxel-based models suffer from large information loss. Motivated by the success of recent point-voxel representation, such as PVCNN, we propose a new convolutional neural network, called Multi Point-Voxel Convolution (MPVConv), for deep learning on point clouds. Integrating both the advantages of voxel and point-based methods, MPVConv can effectively increase the neighboring collection between point-based features and also promote independence among voxel-based features. Moreover, most of the existing approaches aim at solving one specific task, and only a few of them can handle a variety of tasks. Simply replacing the corresponding convolution module with MPVConv, we show that MPVConv can fit in different backbones to solve a wide range of 3D tasks. Extensive experiments on benchmark datasets such as ShapeNet Part, S3DIS and KITTI for various tasks show that MPVConv improves the accuracy of the backbone (PointNet) by up to extbf{36\%}, and achieves higher accuracy than the voxel-based model with up to extbf{34}$ imes$ speedups. In addition, MPVConv outperforms the state-of-the-art point-based models with up to extbf{8}$ imes$ speedups. Notably, our MPVConv achieves better accuracy than the newest point-voxel-based model PVCNN (a model more efficient than PointNet) with lower latency.

Motivation & Objective

• To address the inefficiency of point-based models and information loss in voxel-based models for 3D point cloud learning.
• To design a general-purpose 3D deep learning module that supports diverse tasks like semantic segmentation and 3D detection.
• To integrate the strengths of both point and voxel representations—neighborhood aggregation for points and feature independence for voxels—within a unified, efficient architecture.
• To enable plug-and-play integration with existing backbones like PointNet for broad applicability across 3D vision tasks.

Proposed method

• MPVConv employs a multi-branch architecture with separate 3D CNN and MLP branches for point-based and voxel-based features, respectively.
• It uses 1×1×1 3D convolutions to fuse and enhance cross-modality feature interactions between point and voxel representations.
• The fusion module combines features from both branches, improving feature representation without significant memory or latency overhead.
• The method applies multi-scale feature learning by processing points and voxels at different resolutions, enhancing local geometric understanding.
• It leverages hierarchical feature aggregation to strengthen local neighborhood connectivity in point-based features.
• The architecture is designed to be modular and compatible with various backbones, enabling easy adaptation to different 3D tasks.

Figure 1: MPVConv increases neighborhood collection in point-based features and independence in voxel-based features by adopting 3D CNNs and MLPs on both types of features.

Experimental results

Research questions

• RQ1 Can a unified 3D convolution module effectively combine the strengths of point-based and voxel-based deep learning for improved performance and efficiency?
• RQ2 How does MPVConv enhance neighborhood collection in point-based features while promoting independence in voxel-based features?
• RQ3 To what extent can MPVConv improve accuracy and speed across diverse 3D tasks such as part segmentation, indoor scene segmentation, and 3D detection?
• RQ4 How does MPVConv compare to state-of-the-art models like PVCNN and F-PointNet in terms of accuracy, inference speed, and memory efficiency?
• RQ5 Can MPVConv be seamlessly integrated into existing backbones like PointNet to boost performance across multiple benchmarks?

Key findings

• MPVConv improves the accuracy of the PointNet backbone by up to 36% on the ShapeNet Part dataset, achieving 85.76% mIoU.
• On the S3DIS dataset, MPVConv achieves 79.85% mIoU for large-scale indoor scene segmentation, outperforming F-PVCNN and other baselines.
• For 3D detection on KITTI, F-MPVCNN improves mAP by up to 13.2% over F-PointNet and by 8.6% on the hard pedestrian class compared to F-PointNet++.
• MPVConv achieves up to 34× speedup over voxel-based models while maintaining higher accuracy, demonstrating superior efficiency.
• The 1×1×1 3D CNN layer in MPVConv improves mIoU by 0.04% with only 0.4ms latency increase and 0.009GB memory overhead.
• Ablation studies confirm that the fusion module and combined feature components (V₁ + V₂ + P₂) are critical for performance, with the full MPVConv configuration yielding the best results.

Figure 2: Segmentation results on ShapeNet Part. (a)(c) Ground truth. (b)(d) Our results.