[Paper Review] 3D Siamese Transformer Network for Single Object Tracking on Point Clouds

Le Hui, Lingpeng Wang+4 more|arXiv (Cornell University)|Jul 25, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes STNet, a 3D Siamese Transformer network for single object tracking on point clouds that leverages self-attention in an encoder-decoder architecture to learn shape context and cross-attention for robust template-search correlation. It achieves state-of-the-art performance on KITTI, nuScenes, and Waymo benchmarks by iteratively refining feature fusion through coarse-to-fine correlation with ego-feature and cross-feature augmentation.

ABSTRACT

Siamese network based trackers formulate 3D single object tracking as cross-correlation learning between point features of a template and a search area. Due to the large appearance variation between the template and search area during tracking, how to learn the robust cross correlation between them for identifying the potential target in the search area is still a challenging problem. In this paper, we explicitly use Transformer to form a 3D Siamese Transformer network for learning robust cross correlation between the template and the search area of point clouds. Specifically, we develop a Siamese point Transformer network to learn shape context information of the target. Its encoder uses self-attention to capture non-local information of point clouds to characterize the shape information of the object, and the decoder utilizes cross-attention to upsample discriminative point features. After that, we develop an iterative coarse-to-fine correlation network to learn the robust cross correlation between the template and the search area. It formulates the cross-feature augmentation to associate the template with the potential target in the search area via cross attention. To further enhance the potential target, it employs the ego-feature augmentation that applies self-attention to the local k-NN graph of the feature space to aggregate target features. Experiments on the KITTI, nuScenes, and Waymo datasets show that our method achieves state-of-the-art performance on the 3D single object tracking task.

Motivation & Objective

• To address the challenge of large appearance variation between template and search area in 3D single object tracking on sparse, irregular point clouds.
• To improve feature representation by capturing long-range shape context using a Siamese Transformer encoder-decoder structure.
• To enhance cross-correlation learning between template and search area through iterative coarse-to-fine feature fusion.
• To boost discriminative feature learning by applying self-attention on local k-NN graphs in feature space (ego-feature augmentation).
• To achieve state-of-the-art tracking performance on standard 3D tracking benchmarks using an end-to-end, attention-based framework.

Proposed method

• A Siamese point Transformer network with encoder-decoder architecture uses self-attention in the encoder to capture non-local shape context from point clouds.
• The decoder employs adaptive feature interpolation with learnable attention weights to upsample features from subsampled points to original resolution.
• An iterative coarse-to-fine correlation network applies cross-attention to fuse template and search area features, enabling coarse localization.
• Ego-feature augmentation applies self-attention to local k-NN graphs in feature space to aggregate semantically similar features and enhance target representation.
• The framework jointly optimizes feature learning and correlation via end-to-end training, with attention maps guiding target localization.
• Hyperparameters such as k-NN size (K) and number of iterations are tuned for optimal performance and memory efficiency.

Experimental results

Research questions

• RQ1 Can self-attention mechanisms effectively model long-range shape context in sparse 3D point clouds for improved tracking?
• RQ2 How does iterative coarse-to-fine correlation with cross- and ego-attention improve robustness to appearance variation in tracking?
• RQ3 What is the optimal balance between local feature aggregation (via k-NN) and global attention in 3D tracking?
• RQ4 How does the proposed Siamese Transformer compare to existing 2D and 3D Siamese baselines in terms of accuracy and inference efficiency?
• RQ5 To what extent does shape context learning via attention improve tracking performance on real-world 3D datasets?

Key findings

• The proposed STNet achieves state-of-the-art performance on KITTI, nuScenes, and Waymo datasets, with a success rate of 72.1% and precision of 84.0%.
• Ablation studies show that combining non-local embedding and adaptive feature interpolation improves performance from 66.1% (PointNet++) to 72.1% success rate.
• Using both cross-feature and ego-feature augmentation (CF aug. + EF aug.) yields the best results, outperforming single-module baselines.
• The optimal k-NN size for ego-feature augmentation is K=48, balancing local structure capture and background noise suppression.
• Two iterations in the coarse-to-fine correlation network provide the best trade-off between performance and GPU memory usage.
• Visualization confirms that STNet with all components effectively focuses on the target, while partial models fail to distinguish the car from background.