[Paper Review] Robust Tracking Using Region Proposal Networks

Jimmy Ren, Zhiyang Yu+6 more|arXiv (Cornell University)|May 30, 2017

Video Surveillance and Tracking Methods18 references4 citations

TL;DR

This paper proposes RPN2T, a novel visual tracking method that leverages pre-trained Region Proposal Network (RPN) top-layer features without ensemble or feature engineering. By introducing a novel loss function that jointly optimizes classification accuracy and bounding box quality, the method achieves state-of-the-art performance on OTB50, OTB100, and VOT2016 benchmarks, outperforming existing CNN-based and hand-crafted feature trackers.

ABSTRACT

Recent advances in visual tracking showed that deep Convolutional Neural Networks (CNN) trained for image classification can be strong feature extractors for discriminative trackers. However, due to the drastic difference between image classification and tracking, extra treatments such as model ensemble and feature engineering must be carried out to bridge the two domains. Such procedures are either time consuming or hard to generalize well across datasets. In this paper we discovered that the internal structure of Region Proposal Network (RPN)'s top layer feature can be utilized for robust visual tracking. We showed that such property has to be unleashed by a novel loss function which simultaneously considers classification accuracy and bounding box quality. Without ensemble and any extra treatment on feature maps, our proposed method achieved state-of-the-art results on several large scale benchmarks including OTB50, OTB100 and VOT2016. We will make our code publicly available.

Motivation & Objective

• To eliminate the need for model ensemble and manual feature engineering in CNN-based visual tracking.
• To investigate whether pre-trained RPN top-layer features can serve as strong discriminative representations for visual tracking.
• To discover a loss function that unlocks the full potential of RPN features for tracking by jointly optimizing classification and localization.
• To achieve state-of-the-art performance on standard benchmarks using a single, unified model without domain-specific adaptation.

Proposed method

• Utilizes top-layer feature maps from a pre-trained RPN, originally trained for object detection, as the backbone feature representation for tracking.
• Introduces a novel joint loss function that simultaneously optimizes classification accuracy and bounding box regression quality to better align features with tracking objectives.
• Employs a single RPN-based model for both region proposal and tracking, avoiding the need for multiple networks or feature fusion strategies.
• Applies standard online model update protocols from prior work (e.g., [20]) for long-term and short-term adaptation during inference.
• Uses a lightweight network architecture, significantly smaller than VGG-based models used in prior SOTA trackers.
• Employs standard evaluation protocols on OTB50, OTB100, and VOT2016 with no additional data augmentation or domain-specific tuning.

Experimental results

Research questions

• RQ1 Can the top-layer features of a pre-trained RPN be effectively repurposed for visual tracking without additional training or feature engineering?
• RQ2 What loss function design is necessary to unlock the discriminative potential of RPN features for tracking, given the mismatch between object detection and visual tracking objectives?
• RQ3 Can a single, unified model using RPN features outperform existing CNN-based and hand-crafted feature trackers on standard benchmarks?
• RQ4 Does the proposed method maintain robustness across diverse challenging scenarios such as occlusion, illumination changes, and pose variations?

Key findings

• RPN2T achieved state-of-the-art performance on OTB50 and OTB100, outperforming both hand-crafted feature methods and other CNN-based trackers.
• On VOT2016, RPN2T achieved the highest Expected Average Overlap (EAO) score of 0.335, ranking first among all methods evaluated.
• The method achieved competitive accuracy (ACC = 0.54) and robustness (ROB = 0.87) on VOT2016, outperforming C-COT and TCNN in EAO despite using a smaller network.
• In challenging scenarios on OTB100, RPN2T demonstrated superior robustness across all eight categories, consistently outperforming all baselines.
• The proposed joint loss function was essential—direct use of RPN features without it failed to yield strong results, proving its necessity to unlock feature potential.
• The method achieved ~2 FPS inference speed on a GTX Titan X GPU, demonstrating practical feasibility despite not being optimized for speed.