[Paper Review] Fast Online Object Tracking and Segmentation: A Unifying Approach

Qiang Wang, Li Zhang+3 more|arXiv (Cornell University)|Dec 12, 2018

Video Surveillance and Tracking Methods51 references98 citations

TL;DR

SiamMask extends fully-convolutional Siamese trackers with a binary segmentation branch to jointly perform real-time object tracking and semi-supervised video object segmentation, achieving state-of-the-art real-time tracking on VOT-2018 and competitive, fastest performance on DAVIS-2016/2017.

ABSTRACT

In this paper we illustrate how to perform both visual object tracking and semi-supervised video object segmentation, in real-time, with a single simple approach. Our method, dubbed SiamMask, improves the offline training procedure of popular fully-convolutional Siamese approaches for object tracking by augmenting their loss with a binary segmentation task. Once trained, SiamMask solely relies on a single bounding box initialisation and operates online, producing class-agnostic object segmentation masks and rotated bounding boxes at 55 frames per second. Despite its simplicity, versatility and fast speed, our strategy allows us to establish a new state of the art among real-time trackers on VOT-2018, while at the same time demonstrating competitive performance and the best speed for the semi-supervised video object segmentation task on DAVIS-2016 and DAVIS-2017. The project website is http://www.robots.ox.ac.uk/~qwang/SiamMask.

Motivation & Objective

• Motivate a simple, unified approach for online tracking and video object segmentation.
• Improve target representation by predicting per-frame binary masks in addition to bounding boxes.
• Maintain offline trainability and online speed characteristic of Siamese trackers.
• Demonstrate real-time performance without test-time fine-tuning or heavy augmentations.

Proposed method

• Extend Siamese tracking with a mask branch that predicts a w×h binary mask for each RoW (region of interest window).
• Use depth-wise cross-correlation to produce a dense, multi-channel response map representing candidate similarities.
• Train three task heads offline: similarity (L_sim), mask (L_mask), and bounding-box/score (L_box/L_score) losses, merged into a joint multi-task objective.
• Refine masks via multi-resolution feature fusion with skip connections, following a mask refinement strategy.
• Provide two inference variants: SiamMask-2B (two branches) and SiamMask (three branches) with different reference-box strategies for online tracking.
• Evaluate bounding-box outputs by converting masks to axis-aligned or rotated bounding boxes (Min-max, MBR, or Opt strategies).

Experimental results

Research questions

• RQ1 Can a single offline-trained Siamese network perform online tracking and produce per-frame segmentation masks without test-time fine-tuning?
• RQ2 Does adding a segmentation branch improve tracking accuracy and enable high-quality, pixel-accurate object representations in real time?
• RQ3 How does the mask-based representation impact performance on real-time tracking benchmarks (VOT) and semi-supervised VOS benchmarks (DAVIS, YouTube-VOS)?
• RQ4 What are the trade-offs between different bounding box generation strategies (Min-max, MBR, Opt) when deriving boxes from segmentation masks?

Key findings

• SiamMask achieves real-time operation at 55–60 fps on an RTX 2080 GPU without test-time adaptation.
• On VOT-2018, SiamMask with MBR/Opt representations achieves state-of-the-art performance among real-time trackers, with notable improvements in EAO and accuracy metrics.
• SiamMask is competitive with recent fast VOS approaches on DAVIS-2016/2017 and is the fastest among strong baselines that do not rely on fine-tuning.
• Two variants, SiamMask-2B and SiamMask, demonstrate trade-offs between speed and accuracy, with the 3-branch version offering higher accuracy and the 2-branch version offering faster inference.
• Using the mask branch to inform box generation improves bounding-box-based metrics across benchmarks.