[Paper Review] Proposal, Tracking and Segmentation (PTS): A Cascaded Network for Video Object Segmentation

Qiang Zhou, Zilong Huang+6 more|arXiv (Cornell University)|Jul 2, 2019

Video Surveillance and Tracking Methods39 references4 citations

TL;DR

PTSNet proposes a cascaded video object segmentation framework comprising Object Proposal, Tracking, and Dynamic Reference Segmentation networks. By leveraging objectness for efficient localization, online tracking with scale adaptation, and dynamic reference learning from historical predictions, it achieves state-of-the-art performance on DAVIS’17 and YouTube-VOS, reaching 71.6% JIoU with full training and inference pipeline.

ABSTRACT

Video object segmentation (VOS) aims at pixel-level object tracking given only the annotations in the first frame. Due to the large visual variations of objects in video and the lack of training samples, it remains a difficult task despite the upsurging development of deep learning. Toward solving the VOS problem, we bring in several new insights by the proposed unified framework consisting of object proposal, tracking and segmentation components. The object proposal network transfers objectness information as generic knowledge into VOS; the tracking network identifies the target object from the proposals; and the segmentation network is performed based on the tracking results with a novel dynamic-reference based model adaptation scheme. Extensive experiments have been conducted on the DAVIS'17 dataset and the YouTube-VOS dataset, our method achieves the state-of-the-art performance on several video object segmentation benchmarks. We make the code publicly available at https://github.com/sydney0zq/PTSNet.

Motivation & Objective

• To address the challenge of semi-supervised video object segmentation under large visual variations and limited training data.
• To improve localization accuracy and reduce search space by introducing objectness-aware proposal generation.
• To enhance segmentation robustness through dynamic reference learning using historical predictions as pseudo-ground-truths.
• To decouple localization and segmentation tasks via a modular, cascaded architecture for better feature learning.
• To enable online model adaptation without full fine-tuning, improving generalization under appearance changes.

Proposed method

• An Object Proposal Network (OPN) generates class-agnostic, high-objectness regions to reduce the search space for tracking.
• An Object Tracking Network (OTN) identifies the target object by scoring proposals and adapting online to scale and appearance changes.
• A Dynamic Reference Segmentation Network (DRSN) uses both the first-frame annotation and multiple past segmentation masks as dynamic references to guide current frame prediction.
• The DRSN employs a novel model adaptation scheme that updates the segmentation network using high-confidence historical predictions as pseudo-annotations.
• The framework is end-to-end trainable and causal, enabling real-time inference without future frame dependency.
• Online fine-tuning is applied to further improve performance, especially under long-term occlusions or appearance shifts.

Experimental results

Research questions

• RQ1 Can objectness-aware proposal generation significantly improve localization efficiency and accuracy in video object segmentation?
• RQ2 How does dynamic reference learning from historical predictions compare to static reference or online fine-tuning in segmentation performance?
• RQ3 Can a cascaded architecture that separates proposal, tracking, and segmentation improve robustness to appearance and motion variations?
• RQ4 To what extent does objectness information enhance tracking performance when DAVIS is treated as a tracking benchmark?
• RQ5 How does the integration of online model adaptation via dynamic references affect long-term tracking stability?

Key findings

• PTSNet achieves 71.6% JIoU on the DAVIS’17 validation set, outperforming previous state-of-the-art methods.
• The ablation study shows that adding dynamic references improves performance by 5.1% over static reference (RGMP) alone.
• Introducing the Object Proposal Network (OPN) boosts performance by 2.2% over Gaussian sampling baseline, demonstrating the value of objectness in localization.
• The combination of OPN, OTN, and DRSN with dynamic references achieves 66.1% JIoU, a 7.3% improvement over baseline RGMP.
• Using six reference frames (including N-2, N-4, N-6) yields optimal balance between accuracy (66.1%) and inference speed (56.15ms/frame) on Titan V.
• When evaluated as a tracking dataset, OPN improves AUC from 52.1% (Gaussian sampling) to 77.8%, proving objectness enhances tracking reliability.