[Paper Review] Siamese Instance Search for Tracking

Ran Tao, Efstratios Gavves+1 more|arXiv (Cornell University)|May 19, 2016

Video Surveillance and Tracking MethodsComputer Science49 references143 citations

TL;DR

A tracking approach that uses a pre-trained Siamese network to learn a generic patch-matching function from external videos, and applies it in a simple, non-updating tracker to locate the initial target in subsequent frames. It achieves state-of-the-art performance on OTB without model updates or occlusion handling.

ABSTRACT

In this paper we present a tracker, which is radically different from state-of-the-art trackers: we apply no model updating, no occlusion detection, no combination of trackers, no geometric matching, and still deliver state-of-the-art tracking performance, as demonstrated on the popular online tracking benchmark (OTB) and six very challenging YouTube videos. The presented tracker simply matches the initial patch of the target in the first frame with candidates in a new frame and returns the most similar patch by a learned matching function. The strength of the matching function comes from being extensively trained generically, i.e., without any data of the target, using a Siamese deep neural network, which we design for tracking. Once learned, the matching function is used as is, without any adapting, to track previously unseen targets. It turns out that the learned matching function is so powerful that a simple tracker built upon it, coined Siamese INstance search Tracker, SINT, which only uses the original observation of the target from the first frame, suffices to reach state-of-the-art performance. Further, we show the proposed tracker even allows for target re-identification after the target was absent for a complete video shot.

Motivation & Objective

• Learn a generic, robust patch-matching function from external video data that handles common appearance variations in tracking.
• Develop a simple tracking inference that uses the initial frame's target without per-target adaptation.
• Demonstrate that a well-trained matching function suffices for competitive or state-of-the-art tracking performance.
• Show that the approach supports target re-identification after absence in a video shot.

Proposed method

• Use a two-stream Siamese network (Siamese Invariance Network) to learn a generic matching function for patches.
• Process full images with region-of-interest pooling to efficiently compare multiple candidate patches.
• Fuse features from multiple CNN layers and apply L2 normalization before the loss to stabilize representations.
• Train with a margin contrastive loss D = ||f(x_j) - f(x_k)||_2, y_jk ∈ {0,1}, and margin ε, using external video data (ALOV) with no overlap to evaluation data.
• Employ radius-based sampling to generate candidate boxes around the previous frame prediction and perform box refinement with fixed regressors.
• Track by selecting the candidate patch that maximally matches the initial frame target using m(x_t0, x_jt) = f(x_t0)^T f(x_jt).

Experimental results

Research questions

• RQ1 Can a generically learned patch-matching function, trained on external videos, robustly track unseen targets without online adaptation?
• RQ2 Does a Siamese network-based matching function suffice for accurate frame-to-frame localization when used with a simple, non-updating tracker?
• RQ3 What architectural choices (depth, pooling, multi-layer features) maximize tracking accuracy without target-specific updates?
• RQ4 Can this approach support target re-identification after long-term absence in a video?
• RQ5 How does the proposed method compare to contemporary trackers on standard benchmarks like OTB?

Key findings

• A Siamese network trained on external data significantly improves over ImageNet-pretrained baselines for tracking matching functions.
• Removing max pooling and using multi-layer features (conv4/conv5/fc6) with a deeper network yields the best localization and accuracy.
• SINT achieves state-of-the-art performance on OTB with a simple online inference, and SINT+ further improves with adaptive sampling and optical flow.
• The method demonstrates robust re-identification after target absence within a long video sequence.
• On external YouTube sequences, SINT outperforms MEEM and MUSTer in AUC scores, indicating strong generalization to challenging distortions.