[Paper Review] Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategy

Jiuming Liu, Ruiji Yu+5 more|arXiv (Cornell University)|Mar 11, 2024

Remote Sensing and LiDAR Applications7 citations

TL;DR

Point Mamba introduces a state-space-model–based backbone for point clouds, using an octree-inspired z-order ordering to impose causality, achieving state-of-the-art-like results with linear complexity versus transformers.

ABSTRACT

Recently, state space model (SSM) has gained great attention due to its promising performance, linear complexity, and long sequence modeling ability in both language and image domains. However, it is non-trivial to extend SSM to the point cloud field, because of the causality requirement of SSM and the disorder and irregularity nature of point clouds. In this paper, we propose a novel SSM-based point cloud processing backbone, named Point Mamba, with a causality-aware ordering mechanism. To construct the causal dependency relationship, we design an octree-based ordering strategy on raw irregular points, globally sorting points in a z-order sequence and also retaining their spatial proximity. Our method achieves state-of-the-art performance compared with transformer-based counterparts, with 93.4% accuracy and 75.7 mIOU respectively on the ModelNet40 classification dataset and ScanNet semantic segmentation dataset. Furthermore, our Point Mamba has linear complexity, which is more efficient than transformer-based methods. Our method demonstrates the great potential that SSM can serve as a generic backbone in point cloud understanding. Codes are released at https://github.com/IRMVLab/Point-Mamba.

Motivation & Objective

• Motivate applying a state space model (SSM) backbone to unordered, irregular point clouds.
• Develop a causality-aware ordering mechanism to adapt SSM to point clouds.
• Design a Point Mamba block with bidirectional selective scanning for hierarchical features.
• Demonstrate competitive accuracy and efficiency against transformer-based backbones on ModelNet40 and ScanNet.

Proposed method

• Introduce an octree-based ordering that sorts points along a 3D z-order curve while preserving spatial proximity.
• Discretize and parameterize the SSM to obtain a global, linear-complexity backbone for sequence-like point data.
• Build Point Mamba blocks with bidirectional selective scanning to capture long-range dependencies with minimal complexity.
• Embed point features in a hierarchical architecture with downsampling and a lightweight FPN for downstream tasks.
• Compare against transformer-based backbones and analyze parameters, FLOPs, memory, and speed.

Experimental results

Research questions

• RQ1 Can the SSM backbone effectively model global dependencies in point clouds after imposing a causality-consistent ordering?
• RQ2 Does octree-based z-order ordering preserve spatial proximity while enabling causal dependencies for SSM in point clouds?
• RQ3 How does Point Mamba compare to transformer-based backbones in accuracy and efficiency on ModelNet40 and ScanNet?
• RQ4 What are the memory, parameter, and computation trade-offs of Point Mamba relative to OctFormer and PCT?

Key findings

• Point Mamba (C) achieves 93.4% accuracy on ModelNet40, outperforming the PCT baseline (93.2%).
• Point Mamba (O) achieves 92.7% accuracy on ModelNet40, competitive with OctFormer (92.7%).
• On ScanNet semantic segmentation, Point Mamba reaches 74.6% mIoU without voting and 75.7% with voting (competitive with OctFormer).
• Point Mamba has linear complexity with parameters around 3.08M for classification and 31.99M for segmentation, fewer than OctFormer in both tasks.
• Point Mamba demonstrates lower GPU memory growth (linear with sequence length) and faster forward speed (e.g., 90 ms for Point Mamba-small with voting).
• Point Mamba reduces parameter count by over 25% versus OctFormer while maintaining competitive performance.