[Paper Review] CenterFormer: Center-based Transformer for 3D Object Detection

Zixiang Zhou, Xiangchen Zhao+3 more|arXiv (Cornell University)|Sep 12, 2022

Advanced Neural Network Applications10 citations

TL;DR

CenterFormer proposes a center-based transformer network for 3D object detection in LiDAR point clouds, using center feature embeddings as queries to reduce training difficulty and computational cost. It achieves state-of-the-art performance with 73.7% mAPH on the Waymo validation set and 75.6% on the test set, outperforming prior CNN and transformer-based methods.

ABSTRACT

Query-based transformer has shown great potential in constructing long-range attention in many image-domain tasks, but has rarely been considered in LiDAR-based 3D object detection due to the overwhelming size of the point cloud data. In this paper, we propose CenterFormer, a center-based transformer network for 3D object detection. CenterFormer first uses a center heatmap to select center candidates on top of a standard voxel-based point cloud encoder. It then uses the feature of the center candidate as the query embedding in the transformer. To further aggregate features from multiple frames, we design an approach to fuse features through cross-attention. Lastly, regression heads are added to predict the bounding box on the output center feature representation. Our design reduces the convergence difficulty and computational complexity of the transformer structure. The results show significant improvements over the strong baseline of anchor-free object detection networks. CenterFormer achieves state-of-the-art performance for a single model on the Waymo Open Dataset, with 73.7% mAPH on the validation set and 75.6% mAPH on the test set, significantly outperforming all previously published CNN and transformer-based methods. Our code is publicly available at https://github.com/TuSimple/centerformer

Motivation & Objective

• To address the challenges of training instability and high computational cost in transformer-based 3D object detection on large LiDAR point clouds.
• To improve long-range and contextual feature learning in anchor-free 3D detection by leveraging object-level attention.
• To enable effective multi-frame feature fusion through cross-attention mechanisms for improved detection robustness.
• To replace learnable query embeddings with center feature embeddings to accelerate convergence and improve performance.

Proposed method

• A voxel-based backbone network encodes the LiDAR point cloud into a bird’s-eye view (BEV) feature map.
• A multi-scale center proposal network (CPN) predicts initial center locations and generates center feature embeddings as query inputs.
• A transformer decoder with deformable cross-attention layers aggregates features from multi-scale BEV features using sampled keypoint locations.
• The query embedding is initialized with the feature at the predicted center, enabling focused attention on object-relevant regions.
• Multi-frame feature fusion is achieved via cross-attention between queries and features from temporally adjacent frames.
• The model uses a combination of corner supervision, IoU rectification, and fade augmentation to improve training stability and generalization.

Experimental results

Research questions

• RQ1 Can center-based query initialization improve training convergence and performance in transformer-based 3D object detection on large LiDAR point clouds?
• RQ2 How does using center feature embeddings as queries compare to learnable parametric queries in terms of detection accuracy and training stability?
• RQ3 To what extent can deformable cross-attention reduce computational cost while maintaining or improving feature aggregation in 3D detection?
• RQ4 How effective is multi-frame feature fusion through cross-attention in enhancing detection performance on sequential LiDAR scans?
• RQ5 What is the impact of auxiliary supervision and training strategies like IoU rectification on model convergence and final mAPH?

Key findings

• CenterFormer achieves 73.7% mAPH on the Waymo Open Dataset validation set and 75.6% on the test set, setting a new state-of-the-art for single-model performance.
• Using center features as query embeddings improves mAPH by 1.5% compared to learnable query parameters, demonstrating superior convergence and feature representation.
• The combination of self-attention and cross-attention layers boosts performance to 67.0% mAPH, with deformable cross-attention further increasing it to 67.3%.
• IoU rectification and fade augmentation lead to significant gains, improving mAPH by 1.4% and 0.7% respectively when combined with deformable cross-attention.
• The model trained with a proposal-based strategy (not end-to-end set matching) achieves 68.7% mAPH, outperforming the end-to-end DETR-style training by over 20 percentage points.
• Visualizations confirm that attention mechanisms focus on relevant object regions and neighboring objects of the same class, validating the model's ability to learn contextual and long-range dependencies.