[Paper Review] SwinTrack: A Simple and Strong Baseline for Transformer Tracking

Liting Lin, Heng Fan+3 more|arXiv (Cornell University)|Dec 2, 2021

Video Surveillance and Tracking Methods37 references48 citations

TL;DR

SwinTrack proposes a fully attentional Siamese tracker using Swin Transformer with a lightweight motion token to provide temporal context, achieving state-of-the-art results on multiple tracking benchmarks while maintaining real-time speed.

ABSTRACT

Recently Transformer has been largely explored in tracking and shown state-of-the-art (SOTA) performance. However, existing efforts mainly focus on fusing and enhancing features generated by convolutional neural networks (CNNs). The potential of Transformer in representation learning remains under-explored. In this paper, we aim to further unleash the power of Transformer by proposing a simple yet efficient fully-attentional tracker, dubbed SwinTrack, within classic Siamese framework. In particular, both representation learning and feature fusion in SwinTrack leverage the Transformer architecture, enabling better feature interactions for tracking than pure CNN or hybrid CNN-Transformer frameworks. Besides, to further enhance robustness, we present a novel motion token that embeds historical target trajectory to improve tracking by providing temporal context. Our motion token is lightweight with negligible computation but brings clear gains. In our thorough experiments, SwinTrack exceeds existing approaches on multiple benchmarks. Particularly, on the challenging LaSOT, SwinTrack sets a new record with 0.713 SUC score. It also achieves SOTA results on other benchmarks. We expect SwinTrack to serve as a solid baseline for Transformer tracking and facilitate future research. Our codes and results are released at https://github.com/LitingLin/SwinTrack.

Motivation & Objective

• Motivate the use of full Transformer-based representation learning and fusion for Siamese tracking beyond CNN-based or hybrid frameworks.
• Introduce a lightweight motion token to embed historical target trajectory for temporal robustness.
• Develop a simple, efficient, fully-attentional tracking framework based on the Swin Transformer.
• Demonstrate strong performance across multiple large-scale benchmarks with efficient inference.

Proposed method

• Use Swin Transformer backbone to extract template and search region features (T-tokens and S-tokens).
• Implement a concatenation-based fusion encoder that jointly processes template and search tokens for cross-attention interactions.
• Introduce a motion token (E_motion) encoding a history of target trajectories and fuse it in a cross-attention-based decoder to produce vision-motion representations.
• Adopt a head with IoU-aware classification loss (varifocal loss) and generalized IoU loss for regression, with Hanning window post-processing during inference.
• Train with LaSOT, TrackingNet, GOT-10k, and COCO, using AdamW and careful learning-rate scheduling to stabilize training.

Experimental results

Research questions

• RQ1 Can a fully attentional (Transformer-based) tracker outperform CNN-based and hybrid trackers in representation learning and feature fusion for Siamese tracking?
• RQ2 Does introducing a motion token that encodes historical trajectories improve robustness to distractors and temporal consistency?
• RQ3 Is a simple concatenation-based fusion with a Swin Transformer backbone sufficient to achieve state-of-the-art results across diverse benchmarks?
• RQ4 What is the impact of architectural choices (positional encoding, loss functions, and decoding strategy) on tracking performance and efficiency?

Key findings

• SwinTrack-T-224 achieves 0.672 SUC on LaSOT and about 98 fps, rivaling other Transformer trackers.
• SwinTrack-B-384 sets a new record on LaSOT with 0.713 SUC and achieves strong results across LaSOT ext, TrackingNet, GOT-10k, and TNL2k.
• Motion token improves performance significantly across datasets, especially on LaSOT_ext and GOT-10k, with motion-token-enabled variants outperforming their counterparts without motion tokens.
• The lightweight motion token provides robustness with negligible computation, and ablations show the effectiveness of the embedding-based trajectory representation over a simple learnable token.
• Compared to resnet backbones, Swin Transformer backbones substantially boost SUC scores, and concatenation-based fusion outperforms cross-attention-based fusion in this setting.
• SwinTrack maintains competitive speed (up to 98 fps for the lighter variant) while achieving state-of-the-art accuracy on multiple benchmarks.