[Paper Review] ConDaFormer: Disassembled Transformer with Local Structure Enhancement for 3D Point Cloud Understanding

Lunhao Duan, Shanshan Zhao+4 more|arXiv (Cornell University)|Dec 18, 2023

3D Surveying and Cultural Heritage9 citations

TL;DR

ConDaFormer introduces a disassembled window attention mechanism that splits 3D windows into three orthogonal 2D planes to reduce computation, and adds a local structure enhancement via depth-wise sparse convolutions to capture local geometry for improved 3D point cloud segmentation and detection.

ABSTRACT

Transformers have been recently explored for 3D point cloud understanding with impressive progress achieved. A large number of points, over 0.1 million, make the global self-attention infeasible for point cloud data. Thus, most methods propose to apply the transformer in a local region, e.g., spherical or cubic window. However, it still contains a large number of Query-Key pairs, which requires high computational costs. In addition, previous methods usually learn the query, key, and value using a linear projection without modeling the local 3D geometric structure. In this paper, we attempt to reduce the costs and model the local geometry prior by developing a new transformer block, named ConDaFormer. Technically, ConDaFormer disassembles the cubic window into three orthogonal 2D planes, leading to fewer points when modeling the attention in a similar range. The disassembling operation is beneficial to enlarging the range of attention without increasing the computational complexity, but ignores some contexts. To provide a remedy, we develop a local structure enhancement strategy that introduces a depth-wise convolution before and after the attention. This scheme can also capture the local geometric information. Taking advantage of these designs, ConDaFormer captures both long-range contextual information and local priors. The effectiveness is demonstrated by experimental results on several 3D point cloud understanding benchmarks. Code is available at https://github.com/LHDuan/ConDaFormer .

Motivation & Objective

• Reduce computational cost of self-attention on large-scale 3D point clouds without sacrificing context.
• Incorporate local geometric priors into transformer blocks for better feature learning on irregular point clouds.
• Enable strong performance on 3D semantic segmentation and object detection benchmarks.
• Provide an efficient backbone design for downstream 3D vision tasks.
• Offer ablations to validate the contribution of disassembly and local structure enhancements.

Proposed method

• Disassemble cubic window into three orthogonal 2D planes (XY, XZ, YZ) to compute self-attention with reduced point counts.
• Apply a context-based adaptive relative position encoding across planes to preserve positional information.
• Fuse attention outputs from three planes and project via a linear layer (DaFormer).
• Introduce a Local Structure Enhancement (LSE) module with pre- and post-attention depth-wise sparse convolution and 1x1x1 conv branches.
• Place LSE before and after attention to enrich local context and propagate long-range context to local neighbors.
• Build four-stage backbones with downsampling and use a U-Net style upsampling for semantic segmentation and FCAF3D/CAGroup3D backbones for detection.

Experimental results

Research questions

• RQ1 Can disassembling 3D windows into three orthogonal planes reduce attention computation while maintaining or improving performance?
• RQ2 Does adding local structure enhancement via depth-wise sparse convolution before and after attention improve modeling of local geometry in point clouds?
• RQ3 How does ConDaFormer perform on 3D semantic segmentation and object detection benchmarks compared to state-of-the-art methods?
• RQ4 What ablations confirm the effectiveness of window disassembly and shared position encodings across planes?

Key findings

• Achieves state-of-the-art mIoU on ScanNet v2 and S3DIS with ConDaFormer (and competitive results on ScanNet200).
• DaFormer (disassembled attention) reduces computation compared to cubic window attention, with comparable or improved performance depending on settings.
• Local Structure Enhancement (LSE) before and after attention yields improvements, with the best results when both branches are used.
• On SUN RGB-D detection, ConDaFormer achieves competitive mAP@0.25 and mAP@0.5 with fewer parameters than some baselines.
• Ablations show sharing position embeddings across planes and the impact of window size on performance and efficiency.