[Paper Review] Learning Tracking Representations via Dual-Branch Fully Transformer Networks

Fei Xie, Chunyu Wang+3 more|arXiv (Cornell University)|Dec 5, 2021

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes DualTFR, a Siamese-like dual-branch fully transformer network that learns visual tracking representations directly via self-attention mechanisms without relying on CNNs for feature extraction. By using local and global attention blocks within each branch and a single cross-attention block at the final layer, the model achieves state-of-the-art performance on GOT-10k and VOT2020 benchmarks while maintaining real-time inference at ~40 fps on a single GPU.

ABSTRACT

We present a Siamese-like Dual-branch network based on solely Transformers for tracking. Given a template and a search image, we divide them into non-overlapping patches and extract a feature vector for each patch based on its matching results with others within an attention window. For each token, we estimate whether it contains the target object and the corresponding size. The advantage of the approach is that the features are learned from matching, and ultimately, for matching. So the features are aligned with the object tracking task. The method achieves better or comparable results as the best-performing methods which first use CNN to extract features and then use Transformer to fuse them. It outperforms the state-of-the-art methods on the GOT-10k and VOT2020 benchmarks. In addition, the method achieves real-time inference speed (about $40$ fps) on one GPU. The code and models will be released.

Motivation & Objective

• To investigate whether Vision Transformers can outperform CNN-based feature extractors in visual object tracking by learning features directly from matching.
• To design a lightweight, efficient dual-branch transformer architecture that avoids reliance on CNNs while maintaining high accuracy and real-time speed.
• To validate the effectiveness of pure transformer-based feature learning for tracking, especially in comparison to hybrid 'CNN+Transformer' pipelines.
• To provide empirical insights into block design, pretraining, and attention mechanisms for end-to-end transformer-based tracking.

Proposed method

• Divides template and search images into non-overlapping patches and applies a dual-branch transformer architecture to extract features via self-attention within each branch.
• Uses local attention blocks (LAB) on high-resolution feature maps for early feature extraction and global attention blocks (GAB) on low-resolution maps for long-range dependency modeling.
• Introduces a single cross-attention block (CAB) at the final layer to fuse features between the template and search branches, enhancing discriminative representation.
• Employs a classification head and regression head on top of each token to predict object presence and bounding box size directly.
• Optimizes inference speed by applying global attention only on low-resolution features, reducing FLOPs while preserving accuracy.
• Trains the model from scratch or with ImageNet pretraining to evaluate the impact of pretraining on performance and convergence.

Experimental results

Research questions

• RQ1 Can a fully transformer-based architecture outperform dominant 'CNN+Transformer' methods in visual object tracking?
• RQ2 Is self-attention effective for feature extraction in tracking when applied directly to image patches without CNN supervision?
• RQ3 What is the optimal configuration of local, global, and cross-attention blocks for balancing accuracy and inference speed?
• RQ4 How critical is ImageNet pretraining for a fully transformer-based tracking model compared to training from scratch?
• RQ5 Does attention-based feature fusion (CAB) significantly improve tracking performance over traditional correlation-based fusion?

Key findings

• DualTFR achieves a new state-of-the-art EAO score of 0.528 on the VOT2020 benchmark, outperforming existing methods including STARK and TransT.
• On the GOT-10k benchmark, DualTFR achieves an AO score of 73.5% when pre-trained on ImageNet, significantly outperforming the 51.2% score when using depth-wise correlation instead of cross-attention.
• The model runs at approximately 40 fps on a single NVIDIA 2080Ti GPU, demonstrating real-time inference capability.
• Ablation studies show that using global attention blocks (GAB) improves AO from 42.1% to 46.1%, and adding cross-attention blocks increases performance from 42.1% to 48.5%.
• ImageNet pretraining is crucial: performance drops from 73.5% to 48.5% when training from scratch, indicating that pretraining provides essential inductive bias for the fully transformer pipeline.
• The model with 44.1M parameters and 18.9G FLOPs achieves better accuracy than STARK (42.4M parameters, 18.5G FLOPs) on VOT2020, demonstrating superior efficiency-accuracy trade-off.