[Paper Review] Robust Object Modeling for Visual Tracking

Yidong Cai, Jie Liu+2 more|arXiv (Cornell University)|Aug 9, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes ROMTrack, a novel visual tracking framework that enhances robustness by jointly modeling inherent and hybrid template features using dual-stream attention, while introducing adaptive variation tokens to handle appearance changes. The method achieves state-of-the-art performance on six benchmarks, outperforming prior methods by up to 2.5% in AUC on LaSOT ${}_{\text{ext}}$.

ABSTRACT

Object modeling has become a core part of recent tracking frameworks. Current popular tackers use Transformer attention to extract the template feature separately or interactively with the search region. However, separate template learning lacks communication between the template and search regions, which brings difficulty in extracting discriminative target-oriented features. On the other hand, interactive template learning produces hybrid template features, which may introduce potential distractors to the template via the cluttered search regions. To enjoy the merits of both methods, we propose a robust object modeling framework for visual tracking (ROMTrack), which simultaneously models the inherent template and the hybrid template features. As a result, harmful distractors can be suppressed by combining the inherent features of target objects with search regions' guidance. Target-related features can also be extracted using the hybrid template, thus resulting in a more robust object modeling framework. To further enhance robustness, we present novel variation tokens to depict the ever-changing appearance of target objects. Variation tokens are adaptable to object deformation and appearance variations, which can boost overall performance with negligible computation. Experiments show that our ROMTrack sets a new state-of-the-art on multiple benchmarks.

Motivation & Objective

• Address the limitations of separate and interactive template learning in visual tracking by combining their strengths.
• Mitigate the risk of distractor contamination in hybrid template learning while preserving discriminative feature extraction.
• Improve robustness to object deformation and appearance variations through a novel, computation-efficient token design.
• Achieve state-of-the-art performance on multiple standard visual tracking benchmarks without relying on template updating.

Proposed method

• Propose a dual-template architecture with an inherent template stream to preserve pure target features and a hybrid template stream for mutual feature enhancement with the search region.
• Introduce variation tokens derived from hybrid template features to model dynamic appearance changes during tracking.
• Integrate variation tokens into cross-attention computation to guide attention toward appearance context without increasing computational cost.
• Use a two-stage training strategy: first pre-train on a large-scale dataset, then fine-tune with a consecutive sampling strategy to learn temporal appearance dynamics.
• Employ self-attention in the inherent template stream and cross-attention in the hybrid template stream for bidirectional feature interaction.
• Design a lightweight yet effective variation token mechanism that adapts to object deformation and appearance shifts during inference.

Experimental results

Research questions

• RQ1 Can combining inherent and hybrid template features improve tracking robustness compared to using either approach in isolation?
• RQ2 How effective is the proposed variation token mechanism in modeling appearance variations without increasing computational cost?
• RQ3 Does the consecutive sampling strategy during training enhance the model's ability to capture temporal appearance changes?
• RQ4 To what extent does the proposed framework outperform existing state-of-the-art trackers on standard benchmarks?
• RQ5 Can the variation token design be effectively combined with online template updating to further boost performance?

Key findings

• ROMTrack achieves a new state-of-the-art on six benchmarks, with 69.3% AUC on LaSOT and 48.9% on LaSOT ${}_{\text{ext}}$, outperforming prior methods by up to 2.5% in AUC.
• The ablation study confirms that variation tokens are critical: ROMTrack (w/o vt) performs 0.5% and 0.7% worse on LaSOT and LaSOT ${}_{\text{ext}}$, respectively.
• Consecutive sampling during training improves performance over random sampling, demonstrating its effectiveness in learning temporal appearance dynamics.
• ROMTrack outperforms HTM-400 (trained for 400 epochs) by 0.5% AUC on LaSOT and 2.6% on LaSOT ${}_{\text{ext}}$, proving the superiority of the architecture over longer training alone.
• When combined with online template updating, the variation token design further boosts performance, with HTM-vt-online achieving 0.5% and 0.4% higher AUC on LaSOT and LaSOT ${}_{\text{ext}}$.
• The method is robust to distractors and deformation, as shown by consistent performance gains across diverse tracking challenges, including occlusion and scale variation.