[Paper Review] Point Transformer V2: Grouped Vector Attention and Partition-based Pooling

Xiaoyang Wu, Yixing Lao+3 more|arXiv (Cornell University)|Oct 11, 2022

Remote Sensing and LiDAR Applications191 citations

TL;DR

Point Transformer V2 introduces grouped vector attention and partition-based pooling, improving efficiency and accuracy over Point Transformer, and achieving state-of-the-art results on ScanNet v2, S3DIS, and ModelNet40.

ABSTRACT

As a pioneering work exploring transformer architecture for 3D point cloud understanding, Point Transformer achieves impressive results on multiple highly competitive benchmarks. In this work, we analyze the limitations of the Point Transformer and propose our powerful and efficient Point Transformer V2 model with novel designs that overcome the limitations of previous work. In particular, we first propose group vector attention, which is more effective than the previous version of vector attention. Inheriting the advantages of both learnable weight encoding and multi-head attention, we present a highly effective implementation of grouped vector attention with a novel grouped weight encoding layer. We also strengthen the position information for attention by an additional position encoding multiplier. Furthermore, we design novel and lightweight partition-based pooling methods which enable better spatial alignment and more efficient sampling. Extensive experiments show that our model achieves better performance than its predecessor and achieves state-of-the-art on several challenging 3D point cloud understanding benchmarks, including 3D point cloud segmentation on ScanNet v2 and S3DIS and 3D point cloud classification on ModelNet40. Our code will be available at https://github.com/Gofinge/PointTransformerV2.

Motivation & Objective

• Identify limitations of Point Transformer (PTv1) in deep, high-channel scenarios.
• Propose a parameter-efficient grouped vector attention mechanism with a grouped weight encoding layer.
• Strengthen position encoding for 3D geometry via a position encoding multiplier.
• Develop a partition-based (grid) pooling strategy for better spatial alignment and efficiency.
• Demonstrate improved performance on standard 3D benchmarks and analyze model complexity and latency.

Proposed method

• Introduce grouped vector attention (GVA) with a grouped weight encoding function to reduce parameter count while preserving expressive power.
• Define a grouped linear weight encoding (zeta) to enable inter-group information exchange.
• Enhance position encoding with a position encoding multiplier delta_mul applied to the relation vector.
• Propose partition-based pooling using non-overlapping grid partitions (grid pooling) with simple unpooling.
• Maintain a U-Net backbone with skip connections and neighborhood attention for local 3D point interactions.
• Provide extensive ablations comparing GVA, weight encoding variants, position encoding, and pooling strategies.

Experimental results

Research questions

• RQ1 Does grouped vector attention (GVA) outperform traditional multi-head/self-attention (MSA) in local neighborhood attention for point clouds?
• RQ2 Does the grouped weight encoding function reduce parameter count while maintaining or improving performance compared to standard linear encodings?
• RQ3 Does adding a position encoding multiplier enhance spatial reasoning beyond PTv1’s encoding schemes?
• RQ4 Do partition-based pooling methods improve efficiency and spatial alignment over sampling-based pooling in 3D point clouds?

Key findings

• PTv2 outperforms PTv1 and achieves state-of-the-art results on ScanNet v2, S3DIS, and ModelNet40.
• Neighborhood attention with GVA consistently surpasses both shifted-grid attention and MSA in ablations.
• The grouped linear/weight encoding variant with normalization/activation provides better performance with fewer parameters than naive linear encodings.
• Partition-based grid pooling delivers higher mIoU and faster forward times than sampling-based pooling methods (FPS-kNN, Grid-kNN).
• On ModelNet40, PTv2 achieves 91.6 mAcc and 94.2 OA, marking state-of-the-art performance among reported methods.
• PTv2 reduces parameter growth and maintains efficiency as depth and channel counts increase, compared to PTv1.