[Paper Review] Online Tracking by Learning Discriminative Saliency Map with Convolutional Neural Network

Seunghoon Hong, Tackgeun You+2 more|arXiv (Cornell University)|Feb 24, 2015

Video Surveillance and Tracking MethodsComputer Science42 references518 citations

TL;DR

This paper proposes an online visual tracking method that leverages a pre-trained Convolutional Neural Network (CNN) to learn discriminative saliency maps for accurate target localization and pixel-level segmentation. By back-propagating SVM-identified features through the CNN, the approach generates target-specific saliency maps that enhance appearance modeling, achieving state-of-the-art performance on benchmark datasets with superior tracking accuracy and segmentation capability.

ABSTRACT

We propose an online visual tracking algorithm by learning discriminative saliency map using Convolutional Neural Network (CNN). Given a CNN pre-trained on a large-scale image repository in offline, our algorithm takes outputs from hidden layers of the network as feature descriptors since they show excellent representation performance in various general visual recognition problems. The features are used to learn discriminative target appearance models using an online Support Vector Machine (SVM). In addition, we construct target-specific saliency map by backpropagating CNN features with guidance of the SVM, and obtain the final tracking result in each frame based on the appearance model generatively constructed with the saliency map. Since the saliency map visualizes spatial configuration of target effectively, it improves target localization accuracy and enable us to achieve pixel-level target segmentation. We verify the effectiveness of our tracking algorithm through extensive experiment on a challenging benchmark, where our method illustrates outstanding performance compared to the state-of-the-art tracking algorithms.

Motivation & Objective

• To address the challenge of robust visual tracking under complex real-world conditions such as occlusion, illumination changes, and motion blur.
• To improve target appearance modeling by combining deep CNN features with online discriminative learning via SVM.
• To enable pixel-level target segmentation by constructing target-specific saliency maps through back-propagation of discriminative features.
• To achieve online adaptation of the tracker using incremental learning of the SVM and generative model for sustained tracking performance.

Proposed method

• Utilizes pre-trained CNN features from the last convolutional layer as generic, high-level representations for object appearance.
• Employs online SVM to learn discriminative target-background classification using sequential training samples.
• Generates target-specific saliency maps by back-propagating CNN features relevant to positive SVM samples through the network.
• Aggregates saliency maps from multiple positive samples to form a spatially refined map highlighting discriminative target regions.
• Performs tracking via sequential Bayesian filtering using the saliency map as a likelihood observation model.
• Updates both the online SVM and the generative appearance model in real time for continuous adaptation.

Experimental results

Research questions

• RQ1 Can pre-trained CNN features be effectively used for online visual tracking without fine-tuning the network?
• RQ2 How can discriminative saliency maps be constructed from CNN features to improve localization accuracy?
• RQ3 Can saliency maps derived from SVM decision boundaries enhance tracking robustness under challenging visual conditions?
• RQ4 To what extent does the generative model based on saliency maps improve pixel-level segmentation compared to bounding box-only tracking?

Key findings

• The proposed method achieves state-of-the-art performance on the online tracking benchmark, outperforming existing trackers across all evaluation metrics.
• The full algorithm with saliency map-based generative modeling shows significant improvement over the reduced version relying only on SVM scores.
• The tracker achieves high-precision pixel-wise segmentation, with overlap ratios (IoU) substantially exceeding those of bounding box-based trackers.
• Quantitative results demonstrate consistent superiority across all 11 tracking challenge attributes, including occlusion, illumination variation, and fast motion.
• The success plot over 9 video sequences shows the proposed method achieves the highest AUC score, indicating robust and accurate tracking performance.