[Paper Review] Semantic tracking: Single-target tracking with inter-supervised convolutional networks

Jingjing Xiao, Qiang Lan+2 more|arXiv (Cornell University)|Nov 19, 2016

Video Surveillance and Tracking Methods13 references3 citations

TL;DR

This paper proposes a semantic tracker that jointly performs single-target tracking and object category recognition using a unified convolutional network architecture. By integrating a shared feature extractor with category-specific classification and tracking branches that inter-supervise each other, the tracker achieves state-of-the-art performance on the OTB-100 benchmark, outperforming 38 existing methods with an AUC score of 0.572.

ABSTRACT

This article presents a semantic tracker which simultaneously tracks a single target and recognises its category. In general, it is hard to design a tracking model suitable for all object categories, e.g., a rigid tracker for a car is not suitable for a deformable gymnast. Category-based trackers usually achieve superior tracking performance for the objects of that specific category, but have difficulties being generalised. Therefore, we propose a novel unified robust tracking framework which explicitly encodes both generic features and category-based features. The tracker consists of a shared convolutional network (NetS), which feeds into two parallel networks, NetC for classification and NetT for tracking. NetS is pre-trained on ImageNet to serve as a generic feature extractor across the different object categories for NetC and NetT. NetC utilises those features within fully connected layers to classify the object category. NetT has multiple branches, corresponding to multiple categories, to distinguish the tracked object from the background. Since each branch in NetT is trained by the videos of a specific category or groups of similar categories, NetT encodes category-based features for tracking. During online tracking, NetC and NetT jointly determine the target regions with the right category and foreground labels for target estimation. To improve the robustness and precision, NetC and NetT inter-supervise each other and trigger network adaptation when their outputs are ambiguous for the same image regions (i.e., when the category label contradicts the foreground/background classification). We have compared the performance of our tracker to other state-of-the-art trackers on a large-scale tracking benchmark (100 sequences)---the obtained results demonstrate the effectiveness of our proposed tracker as it outperformed other 38 state-of-the-art tracking algorithms.

Motivation & Objective

• Address the limitation of category-free trackers that rely on heuristic assumptions like appearance and motion consistency.
• Overcome the lack of generalization in category-based trackers that are limited to specific object types.
• Develop a unified framework that leverages both generic and category-specific features for robust, generalizable tracking.
• Enable online joint classification and tracking without requiring prior knowledge of the target category at initialization.
• Improve tracking robustness through mutual supervision between classification and tracking networks during inference.

Proposed method

• Employ a shared convolutional network (NetS) pre-trained on ImageNet to extract generic features across object categories.
• Use a classification network (NetC) to predict the object category from features extracted by NetS.
• Deploy a multi-branch tracking network (NetT), where each branch is trained on videos of a specific category to learn category-specific foreground-background features.
• During online tracking, NetC identifies the target category and activates the corresponding NetT branch for fine-tuning using foreground and background samples from the first frame.
• Jointly use NetC and NetT outputs to select image regions with consistent category and foreground labels for target localization.
• Trigger network adaptation when NetC and NetT outputs contradict each other, enhancing robustness under ambiguous conditions.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively combine generic and category-specific features for single-target tracking?
• RQ2 How does inter-supervision between classification and tracking networks improve tracking robustness and accuracy?
• RQ3 To what extent can a semantic tracker generalize across diverse object categories while maintaining high performance?
• RQ4 Does joint learning of category recognition and tracking reduce reliance on heuristic assumptions like appearance consistency?
• RQ5 How does the tracker perform under challenging visual conditions such as deformation, illumination changes, and occlusion?

Key findings

• The proposed semantic tracker achieved an AUC score of 0.572 on the OTB-100 benchmark, outperforming 38 state-of-the-art tracking algorithms.
• On sub-datasets with specific attributes—deformation (DEF), illumination variation (IV), scale variation (SV), fast motion (FM), out-of-view (OV), and low resolution (LR)—the tracker achieved the highest AUC scores in all cases.
• The tracker achieved the second-highest AUC in the illumination variation (IV) and in-plane rotation (OPR) sub-datasets, with scores of 0.577 and 0.544 respectively.
• The tracker outperformed the second-best method (HCF) in 9 out of 12 sub-datasets, including the most challenging sub-datasets like deformation and fast motion.
• Failure cases were primarily observed in camouflage scenarios, where the target's appearance closely matches the background, indicating a need for contextual scene understanding.
• The inter-supervision mechanism effectively triggered network adaptation when NetC and NetT outputs conflicted, improving robustness in ambiguous regions.