[Paper Review] VastTrack: Vast Category Visual Object Tracking

Liang Peng, Junyuan Gao+6 more|arXiv (Cornell University)|Mar 6, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

VastTrack introduces a large-scale visual object tracking benchmark with 2,115 object categories and 50,610 video sequences, significantly expanding the scope of existing datasets. It enables training and evaluation of general-purpose trackers through rich annotations, including bounding boxes and linguistic descriptions, demonstrating that current trackers underperform on this diverse, large-scale benchmark, highlighting the need for improved generalization in visual tracking.

ABSTRACT

In this paper, we introduce a novel benchmark, dubbed VastTrack, towards facilitating the development of more general visual tracking via encompassing abundant classes and videos. VastTrack possesses several attractive properties: (1) Vast Object Category. In particular, it covers target objects from 2,115 classes, largely surpassing object categories of existing popular benchmarks (e.g., GOT-10k with 563 classes and LaSOT with 70 categories). With such vast object classes, we expect to learn more general object tracking. (2) Larger scale. Compared with current benchmarks, VastTrack offers 50,610 sequences with 4.2 million frames, which makes it to date the largest benchmark regarding the number of videos, and thus could benefit training even more powerful visual trackers in the deep learning era. (3) Rich Annotation. Besides conventional bounding box annotations, VastTrack also provides linguistic descriptions for the videos. The rich annotations of VastTrack enables development of both the vision-only and the vision-language tracking. To ensure precise annotation, all videos are manually labeled with multiple rounds of careful inspection and refinement. To understand performance of existing trackers and to provide baselines for future comparison, we extensively assess 25 representative trackers. The results, not surprisingly, show significant drops compared to those on current datasets due to lack of abundant categories and videos from diverse scenarios for training, and more efforts are required to improve general tracking. Our VastTrack and all the evaluation results will be made publicly available https://github.com/HengLan/VastTrack.

Motivation & Objective

• To address the limited generalization of existing visual trackers due to insufficient diversity in object categories and training sequences.
• To develop a benchmark that supports both vision-only and vision-language tracking by including linguistic descriptions for videos.
• To enable more faithful evaluation of general tracking performance by covering a vastly broader range of object categories than prior datasets.
• To provide a scalable, high-quality dataset that supports training of robust, generalizable deep learning-based trackers in the era of large models.
• To identify performance gaps in current trackers when evaluated on a diverse, large-scale benchmark, thus motivating future research in generalization.

Proposed method

• The VastTrack benchmark was constructed by curating 50,610 video sequences from diverse real-world scenarios, covering 2,115 distinct object categories.
• All videos were manually annotated with precise bounding boxes through multiple rounds of inspection to ensure high annotation quality.
• Linguistic descriptions were added for each video to support vision-language tracking, enhancing the dataset's utility for multimodal learning.
• The dataset emphasizes short-term tracking with an average sequence length of 83 frames, though it remains applicable for long-term tracking research.
• The benchmark includes comprehensive statistics on object motion, size variation, and IoU between adjacent frames to characterize tracking challenges.
• Evaluation protocols were established to assess 25 state-of-the-art trackers across 10 attributes, using precision, normalized precision, and success metrics.

Experimental results

Research questions

• RQ1 How does the performance of existing visual trackers degrade when evaluated on a benchmark with vastly more object categories and sequences than current standards?
• RQ2 To what extent do vision-language annotations improve the robustness and generalization of visual tracking models on diverse, real-world video sequences?
• RQ3 What are the key challenges in tracking that emerge when testing on a dataset with 2,115 object categories and 50K+ sequences compared to smaller benchmarks?
• RQ4 How do rare object categories (previously unseen in major benchmarks) affect tracker performance, and what factors influence their difficulty?
• RQ5 Can the inclusion of linguistic descriptions in a large-scale tracking benchmark lead to more robust and generalizable tracking models?

Key findings

• Current state-of-the-art trackers show significant performance drops on VastTrack compared to existing benchmarks, indicating poor generalization to unseen categories and diverse scenarios.
• The average success score (SUC) for rare classes like Aardwolf is 0.689, which exceeds that of some common classes such as Actor (0.691), showing that difficulty is not solely determined by category frequency.
• Performance on rare classes such as Air Hockey is very low (avgSUC = 0.052), highlighting the challenge of generalization to infrequent or visually complex objects.
• The dataset's high diversity in object motion, size variation, and rapid changes in IoU between frames contributes to the difficulty of tracking, as evidenced by low performance on attributes like fast motion and large scale variation.
• The inclusion of linguistic descriptions enables new research directions in vision-language tracking, though current models show limited performance on this richly annotated benchmark.
• The benchmark's scale—50,610 sequences and 2,115 categories—surpasses existing datasets like GOT-10k (563 categories) and TrackingNet (27 categories), making it the largest known visual tracking benchmark.