[Paper Review] On Duality Of Multiple Target Tracking and Segmentation

Yicong Tian, Mubarak Shah|arXiv (Cornell University)|Oct 14, 2016

Video Surveillance and Tracking Methods30 references3 citations

TL;DR

This paper proposes a dual optimization framework that jointly solves online multiple target tracking and video object segmentation using Lagrange dual decomposition, enabling mutual improvement: segmentation provides precise foreground masks to reduce tracking drift and ID-switches, while tracking guides segmentation with temporal consistency. The method achieves state-of-the-art performance on both tasks across multiple benchmarks.

ABSTRACT

Traditionally, object tracking and segmentation are treated as two separate problems and solved independently. However, in this paper, we argue that tracking and segmentation are actually closely related and solving one should help the other. On one hand, the object track, which is a set of bounding boxes with one bounding box in every frame, would provide strong high-level guidance for the target/background segmentation task. On the other hand, the object segmentation would separate object from other objects and background, which will be useful for determining track locations in every frame. We propose a novel framework which combines online multiple target tracking and segmentation in a video. In our approach, the tracking and segmentation problems are coupled by Lagrange dual decomposition, which leads to more accurate segmentation results and also \emph{helps resolve typical difficulties in multiple target tracking, such as occlusion handling, ID-switch and track drifting}. To track targets, an individual appearance model is learned for each target via structured learning and network flow is employed to generate tracks from densely sampled candidates. For segmentation, multi-label Conditional Random Field (CRF) is applied to a superpixel based spatio-temporal graph in a segment of video to assign background or target labels to every superpixel. The experiments on diverse sequences show that our method outperforms state-of-the-art approaches for multiple target tracking as well as segmentation.

Motivation & Objective

• Address the limitations of treating tracking and segmentation as separate problems, where coarse bounding boxes in tracking lead to feature contamination and drift.
• Leverage the strong correlation between object tracks (bounding boxes) and pixel-level segmentation to improve both tasks simultaneously.
• Resolve common tracking challenges such as occlusion, ID-switches, and track drifting by incorporating pixel-wise segmentation as a supervisory signal.
• Develop a unified optimization framework that couples tracking and segmentation through dual decomposition, enabling iterative refinement.
• Achieve end-to-end online performance without relying on pre-trained detectors, using structured learning for adaptive appearance models.

Proposed method

• Use Lagrange dual decomposition to couple the tracking and segmentation subproblems, with dual variables enforcing consistency between bounding boxes and pixel labels.
• For tracking, learn individual discriminative appearance models per target via structured learning and generate tracks using network flow on densely sampled candidates.
• For segmentation, construct foreground GMMs per target and a universal background GMM to generate confidence maps, then apply multi-label CRF on a superpixel-based spatio-temporal graph.
• Iteratively optimize both subproblems: segmentation refines track proposals via pixel-level labels, and tracking improves segmentation via consistent bounding box constraints.
• Employ online learning to adapt appearance models during inference, enabling robustness to appearance changes and occlusions.
• Use human-provided detections for initialization in some cases, but allow automatic target discovery via the joint optimization.

Experimental results

Research questions

• RQ1 Can joint optimization of tracking and segmentation improve performance on both tasks compared to solving them independently?
• RQ2 How can pixel-level segmentation results reduce tracking errors such as ID-switches and drift during occlusion?
• RQ3 To what extent can tracking constraints improve the accuracy and temporal consistency of video segmentation?
• RQ4 Can dual decomposition effectively couple tracking and segmentation without requiring pre-trained detectors?
• RQ5 Does the mutual refinement between coarse bounding box tracking and fine-grained segmentation lead to better generalization on challenging sequences?

Key findings

• The proposed method achieves state-of-the-art performance on both multiple target tracking and video object segmentation, outperforming prior methods on PETS-S2L1, TUD-Crossing, and TUD-Stadtmitte.
• On the PETS-S2L1 dataset, the method achieves a MOTA of 92.5 and IDS of 0, significantly outperforming previous trackers and reducing identity switches by a large margin.
• On TUD-Crossing, the method achieves a MOTA of 91.7 and IDS of 0, demonstrating robustness to occlusions and complex scenes.
• On TUD-Stadtmitte, the method achieves a MOTA of 83.8 and IDS of 0, showing strong performance even in crowded environments.
• The joint optimization reduces track drifting and ID-switches by leveraging segmentation to maintain identity consistency during occlusions.
• The ablation study shows that incorporating segmentation improves the tracking-only baseline (Ours-Track) significantly, confirming the mutual benefit between the two tasks.