[Paper Review] Point Transformer V3: Simpler, Faster, Stronger

Xiaoyang Wu, Li Jiang+7 more|arXiv (Cornell University)|Dec 15, 2023

3D Surveying and Cultural HeritageEarth and Planetary Sciences18 citations

TL;DR

Point Transformer V3 (PTv3) achieves scalable, efficient 3D perception by replacing exact neighbor search with serialized point neighborhoods, expanding receptive field from 16 to 1024 while boosting speed and reducing memory, attaining state-of-the-art results on 20+ tasks with multi-dataset training.

ABSTRACT

This paper is not motivated to seek innovation within the attention mechanism. Instead, it focuses on overcoming the existing trade-offs between accuracy and efficiency within the context of point cloud processing, leveraging the power of scale. Drawing inspiration from recent advances in 3D large-scale representation learning, we recognize that model performance is more influenced by scale than by intricate design. Therefore, we present Point Transformer V3 (PTv3), which prioritizes simplicity and efficiency over the accuracy of certain mechanisms that are minor to the overall performance after scaling, such as replacing the precise neighbor search by KNN with an efficient serialized neighbor mapping of point clouds organized with specific patterns. This principle enables significant scaling, expanding the receptive field from 16 to 1024 points while remaining efficient (a 3x increase in processing speed and a 10x improvement in memory efficiency compared with its predecessor, PTv2). PTv3 attains state-of-the-art results on over 20 downstream tasks that span both indoor and outdoor scenarios. Further enhanced with multi-dataset joint training, PTv3 pushes these results to a higher level.

Motivation & Objective

• Motivate scaling as the key driver of performance in 3D point cloud backbones, challenging the notion that intricate module design is always superior.
• Develop a simpler, more efficient backbone by replacing costly KNN-based and relative positional components with serialization-based approaches.
• Increase the effective receptive field dramatically (16 to 1024 points) while maintaining efficiency.
• Demonstrate state-of-the-art results across diverse indoor and outdoor perception tasks, aided by multi-dataset joint training.

Proposed method

• Transform point clouds into a structured sequence via point cloud serialization using space-filling curves (Z-order, Trans Z-order, Hilbert, Trans Hilbert).
• Use serialized encoding to assign a 64-bit code per point and sort within each batch to impose order.
• Adopt patch attention with patch grouping and interaction to broaden the receptive field without dense neighbor search.
• Introduce various patch interaction strategies (Shift Dilation, Shift Patch, Shift Order, Shuffle Order) to enable cross-patch information flow.
• Replace expensive relative positional encoding with an efficient xCPE (enhanced conditional positional encoding) prepended before attention, with optional skip connections.
• Maintain a U-Net-like architecture with four stage encoders/decoders and Grid Pooling, using pre-norm and LayerNorm for stability.

Experimental results

Research questions

• RQ1 Does scaling (data/model) override intricate design choices in 3D backbones, enabling simpler architectures?
• RQ2 Can serialization-based point processing replace KNN and complex neighborhood mechanisms without sacrificing performance?
• RQ3 How does expanding the attention receptive field affect accuracy and efficiency for indoor/outdoor 3D tasks?
• RQ4 What is the impact of multi-dataset joint training on PTv3’s performance?
• RQ5 Which serialization patterns and patch-interaction designs best balance accuracy and efficiency?

Key findings

• PTv3 delivers faster inference (about 3.3x) and lower memory usage (about 10x) compared with PTv2.
• PTv3 expands the attention receptive field from 16 to 1024 points while maintaining efficiency.
• PTv3 achieves state-of-the-art results across more than 20 indoor and outdoor 3D perception tasks.
• Multi-dataset joint training with PTv3 further elevates performance on downstream tasks.
• Serialization-based design, combined with patch attention and xCPE, provides scalable performance gains with negligible additional computational burden when using multiple serialization patterns.