[Paper Review] Transferring Rich Feature Hierarchies for Robust Visual Tracking

Naiyan Wang, Siyi Li+2 more|arXiv (Cornell University)|Jan 19, 2015

Video Surveillance and Tracking Methods37 references290 citations

TL;DR

This paper proposes SO-DLT, a deep learning tracker that transfers rich, pre-trained CNN features for robust visual tracking. By pre-training a CNN to localize objects (not reconstruct images) and using a pixel-wise probability map output, the method enables structured, scalable tracking with online fine-tuning, achieving a 0.602 AUC on a benchmark—10%+ improvement over state-of-the-art trackers.

ABSTRACT

Convolutional neural network (CNN) models have demonstrated great success in various computer vision tasks including image classification and object detection. However, some equally important tasks such as visual tracking remain relatively unexplored. We believe that a major hurdle that hinders the application of CNN to visual tracking is the lack of properly labeled training data. While existing applications that liberate the power of CNN often need an enormous amount of training data in the order of millions, visual tracking applications typically have only one labeled example in the first frame of each video. We address this research issue here by pre-training a CNN offline and then transferring the rich feature hierarchies learned to online tracking. The CNN is also fine-tuned during online tracking to adapt to the appearance of the tracked target specified in the first video frame. To fit the characteristics of object tracking, we first pre-train the CNN to recognize what is an object, and then propose to generate a probability map instead of producing a simple class label. Using two challenging open benchmarks for performance evaluation, our proposed tracker has demonstrated substantial improvement over other state-of-the-art trackers.

Motivation & Objective

• To address the lack of labeled training data in visual tracking, where only one annotated frame is typically available per video.
• To overcome the limitations of prior CNN-based trackers that rely on image reconstruction or classification, which are ill-suited for tracking's structured, localization-based task.
• To improve tracking robustness under appearance variations (e.g., illumination, scale, occlusion) by leveraging transfer learning of rich, object-localizing features.
• To enable effective online adaptation through fine-tuning of the pre-trained CNN during tracking, minimizing drift and overfitting.
• To develop a structured output CNN that predicts pixel-wise object probabilities, capturing spatial consistency better than classification or regression.

Proposed method

• Pre-train a CNN on ImageNet detection data to learn object localization, not image reconstruction, to build a strong inductive bias for objectness.
• Use a fully convolutional network to output a pixel-wise probability map indicating the likelihood of each pixel belonging to the target object.
• Apply a structured loss function based on the pixel-wise map to preserve spatial structure and improve localization accuracy.
• Transfer the pre-trained CNN to online tracking and fine-tune it in real time using the first frame's bounding box and subsequent frames.
• Implement differentially-paced fine-tuning to stabilize updates and reduce drift during occlusion or appearance changes.
• Use the pixel-wise map for inference, enabling robust detection even under large appearance variations.

Experimental results

Research questions

• RQ1 Can pre-trained CNN features trained on large-scale detection data be effectively transferred to visual tracking with minimal labeled data?
• RQ2 Does using a pixel-wise probability map as output improve tracking robustness compared to classification or regression-based approaches?
• RQ3 Can online fine-tuning of a pre-trained CNN mitigate model drift and adapt to appearance changes during tracking?
• RQ4 How does the proposed method perform under extreme appearance variations such as occlusion, illumination changes, and large scale or rotation shifts?
• RQ5 Can the tracker maintain accuracy on non-rigidly deforming objects where bounding box overlap is a poor metric?

Key findings

• The proposed SO-DLT tracker achieves an AUC of 0.602 on the standard benchmark, a significant improvement over the previous state-of-the-art tracker (0.529).
• On the non-rigid object tracking dataset, SO-DLT achieves a mean central pixel error of 21.69, outperforming TGPR (77.88) and PixelTracker (79.26).
• SO-DLT successfully tracks targets through challenging sequences involving large illumination changes, out-of-plane rotations, and scale variations, with minimal drift.
• The tracker maintains robust performance even after occlusion, thanks to differentially-paced fine-tuning that corrects drift once the target reappears.
• Visual results show consistent tracking performance across diverse scenarios, including highly deformable objects like divers and skiers.
• Failure cases are primarily due to distractors with similar appearance or imprecise initial bounding boxes, suggesting room for improvement in feature invariance and initialization.