[Paper Review] Learning to Track Any Object

Achal Dave, Pavel Tokmakov+2 more|arXiv (Cornell University)|Oct 25, 2019

Video Surveillance and Tracking Methods49 references4 citations

TL;DR

This paper proposes a unified framework that repurposes category-specific object detectors into a category-agnostic, object-specific tracker using a learnable objectness prior. By jointly training on image and video data and applying a differentiable, on-the-fly linear template adaptation, the method achieves state-of-the-art tracking and video object segmentation performance with high-quality instance masks, even using only box-level supervision in the first frame.

ABSTRACT

Object tracking can be formulated as "finding the right object in a video". We observe that recent approaches for class-agnostic tracking tend to focus on the "finding" part, but largely overlook the "object" part of the task, essentially doing a template matching over a frame in a sliding-window. In contrast, class-specific trackers heavily rely on object priors in the form of category-specific object detectors. In this work, we re-purpose category-specific appearance models into a generic objectness prior. Our approach converts a category-specific object detector into a category-agnostic, object-specific detector (i.e. a tracker) efficiently, on the fly. Moreover, at test time the same network can be applied to detection and tracking, resulting in a unified approach for the two tasks. We achieve state-of-the-art results on two recent large-scale tracking benchmarks (OxUvA and GOT, using external data). By simply adding a mask prediction branch, our approach is able to produce instance segmentation masks for the tracked object. Despite only using box-level information on the first frame, our method outputs high-quality masks, as evaluated on the DAVIS '17 video object segmentation benchmark.

Motivation & Objective

• To address the challenge of generic, user-initialized object tracking by incorporating a generic objectness prior derived from category-specific detectors.
• To improve robustness against distractors and appearance variations by learning discriminative object templates in a differentiable, end-to-end manner.
• To unify object detection, single-object tracking, and video object segmentation under a single architecture with minimal supervision.
• To achieve state-of-the-art performance on large-scale tracking and video segmentation benchmarks without requiring fine-tuning on test sequences.

Proposed method

• Repurpose a category-specific Mask R-CNN detector into a category-agnostic tracker by training jointly on image (COCO) and video (OxUvA, GOT10k) datasets.
• Introduce a differentiable, closed-form linear discriminator (T_d) that separates the object of interest from background and distractors using features from the first frame.
• Use the template embedding to compute similarity maps that reweight features in subsequent frames, enabling precise localization of the tracked object.
• Extend the model with a mask prediction head to enable instance and video object segmentation with only box-level supervision in the first frame.
• Apply a simple temporal heuristic at test time to maintain tracking consistency across long sequences.
• Train and deploy the same network for detection, tracking, and segmentation, enabling a unified pipeline.

Experimental results

Research questions

• RQ1 Can a category-specific detector be effectively repurposed into a generic, object-specific tracker without requiring category-specific priors?
• RQ2 How can a differentiable, on-the-fly template adaptation strategy improve robustness to distractors and appearance variations in tracking?
• RQ3 To what extent can a unified model trained for detection also achieve state-of-the-art performance in tracking and video object segmentation?
• RQ4 Can high-quality instance segmentation masks be generated from only box-level supervision in the first frame using a unified architecture?

Key findings

• The method achieves state-of-the-art performance on the OxUvA benchmark for long-term tracking, outperforming prior methods without external data.
• On the GOT10k benchmark, the method outperforms state-of-the-art approaches that use external data, achieving a significant margin in tracking accuracy.
• On the LTB35 dataset (VOT 2018 Long-term challenge), the method achieves an F-measure of 61.2 using a single model and hyperparameters across all experiments, outperforming prior methods that require dataset-specific tuning.
• On the DAVIS’17 video segmentation benchmark, the method achieves a mean F-measure of 67.8 and a mean Jaccard of 59.2, performing competitively with specialized methods that require pixel-level mask supervision or end-to-end fine-tuning.
• Despite using only box-level annotations in the first frame, the method produces high-quality segmentation masks, with a decay of 12.0 for F-measure, indicating strong robustness to appearance changes.
• The model achieves a COCO instance segmentation mAP of 30.5 when used as a detector, demonstrating strong generalization despite being optimized for tracking.