[Paper Review] Do Different Tracking Tasks Require Different Appearance Models?

Zhongdao Wang, Hengshuang Zhao+4 more|arXiv (Cornell University)|Jul 5, 2021

Video Surveillance and Tracking Methods110 references32 citations

TL;DR

UniTrack presents a single, task-agnostic appearance model with task-specific heads to tackle five tracking tasks (SOT, VOS, MOT, MOTS, PoseTrack) using propagation and association primitives, enabling competitive results with non-specialized training.

ABSTRACT

Tracking objects of interest in a video is one of the most popular and widely applicable problems in computer vision. However, with the years, a Cambrian explosion of use cases and benchmarks has fragmented the problem in a multitude of different experimental setups. As a consequence, the literature has fragmented too, and now novel approaches proposed by the community are usually specialised to fit only one specific setup. To understand to what extent this specialisation is necessary, in this work we present UniTrack, a solution to address five different tasks within the same framework. UniTrack consists of a single and task-agnostic appearance model, which can be learned in a supervised or self-supervised fashion, and multiple ``heads'' that address individual tasks and do not require training. We show how most tracking tasks can be solved within this framework, and that the same appearance model can be successfully used to obtain results that are competitive against specialised methods for most of the tasks considered. The framework also allows us to analyse appearance models obtained with the most recent self-supervised methods, thus extending their evaluation and comparison to a larger variety of important problems.

Motivation & Objective

• Motivate a unified, task-agnostic appearance representation for multiple tracking tasks.
• Show that a shared representation can achieve competitive results across SOT, VOS, MOT, MOTS, and PoseTrack.
• Introduce propagation and association primitives that work with a common appearance model.
• Propose a reconstruction-based similarity metric for robust association across formats (box, mask, pose).
• Evaluate self-supervised representations as general-purpose features for video tracking.

Proposed method

• Use a trainable, general appearance model φ to extract high-resolution feature maps from frames.
• Express tracking tasks via propagation (SOT, VOS) or association (MOT, MOTS, PoseTrack) primitives.
• Propagate masks, boxes, or poses with appropriate mechanisms (affinity-based mask propagation; cross-correlation/DCF for boxes; Gaussian belief maps for poses).
• Employ a novel Reconstruction Similarity Metric (RSM) to compute symmetric affinities for association across different observation formats.
• Attach task-specific heads that operate on the primitives to solve five tracking tasks without task-specific appearance model training.
• Evaluate a wide range of self-supervised representations (image-level, pixel-level, video-based) within the UniTrack framework.

Experimental results

Research questions

• RQ1 Can a single, shared appearance model perform competitively across five distinct tracking tasks?
• RQ2 How do propagation and association primitives interact with a unified representation to solve SOT, VOS, MOT, MOTS, and PoseTrack?
• RQ3 Does a reconstruction-based similarity measure improve multi-object association across different observation formats?
• RQ4 What is the impact of self-supervised versus supervised representations when used in a unified tracking framework?
• RQ5 To what extent do self-supervised video-based representations outperform image-based ones in video tracking within UniTrack?

Key findings

• A single appearance model can achieve competitive performance across SOT, VOS, MOT, MOTS, and PoseTrack when combined with appropriate task heads.
• A reconstruction-based similarity metric (RSM) preserves fine-grained features and improves association across boxes, masks, and poses.
• Self-supervised representations, especially video-based ones like VFS, generally rank first across tasks, though vanilla ImageNet supervision remains strong in many settings.
• ImageNet-supervised features are surprisingly effective across tasks, providing competitive baselines against specialized methods.
• Video-trained SSL models (e.g., VFS) benefit motion-rich, video-domain tasks, reducing domain gap to downstream tracking tasks.