[Paper Review] Saliency-Enhanced Robust Visual Tracking

Çağlar Aytekin, Francesco Cricri+1 more|arXiv (Cornell University)|Feb 8, 2018

Video Surveillance and Tracking Methods12 references3 citations

TL;DR

This paper proposes ECOSAL, a saliency-enhanced visual tracker that integrates deep salient object detection features into a discrete correlation filter (DCF)-based framework. By adaptively weighting saliency responses based on temporal consistency, the method improves robustness to viewpoint and appearance changes, achieving state-of-the-art performance at 9.3 fps, with superior accuracy on challenging sequences in OTB-2013 and UAV-123 benchmarks.

ABSTRACT

Discrete correlation filter (DCF) based trackers have shown considerable success in visual object tracking. These trackers often make use of low to mid level features such as histogram of gradients (HoG) and mid-layer activations from convolution neural networks (CNNs). We argue that including semantically higher level information to the tracked features may provide further robustness to challenging cases such as viewpoint changes. Deep salient object detection is one example of such high level features, as it make use of semantic information to highlight the important regions in the given scene. In this work, we propose an improvement over DCF based trackers by combining saliency based and other features based filter responses. This combination is performed with an adaptive weight on the saliency based filter responses, which is automatically selected according to the temporal consistency of visual saliency. We show that our method consistently improves a baseline DCF based tracker especially in challenging cases and performs superior to the state-of-the-art. Our improved tracker operates at 9.3 fps, introducing a small computational burden over the baseline which operates at 11 fps.

Motivation & Objective

• Improve the robustness of DCF-based visual trackers in challenging scenarios such as viewpoint changes and appearance variations.
• Address the limitation of low- and mid-level features (e.g., HoG, CNN activations) in handling large appearance and viewpoint changes.
• Leverage semantically higher-level information from deep salient object detection to enhance tracking reliability.
• Develop an adaptive fusion mechanism that dynamically weights saliency features based on their temporal consistency.
• Maintain real-time performance by integrating fast, accurate saliency detection without significant computational overhead.

Proposed method

• Integrate deep salient object detection outputs as an additional feature stream in the ECO tracker framework.
• Learn separate correlation filters for saliency features and traditional features (HoG, VGG-16 conv1/conv5), treating them independently.
• Apply an adaptive weighting mechanism that adjusts the influence of saliency features based on their temporal consistency over time.
• Extract saliency features from the full search region but use a larger context for saliency prediction to improve localization accuracy.
• Use a confidence-based metric to suppress inconsistent saliency responses during tracking, reducing noise from unreliable detections.
• Update the tracker using the combined filter responses, with the adaptive weight dynamically adjusted per frame based on saliency stability.

Experimental results

Research questions

• RQ1 Can integrating deep salient object detection features improve the robustness of DCF-based trackers in challenging visual tracking scenarios?
• RQ2 How can saliency features—known for their semantic robustness—be effectively fused with low- and mid-level features in a DCF framework?
• RQ3 What is the impact of temporal inconsistency in saliency maps on tracker performance, and how can it be mitigated?
• RQ4 Does the proposed adaptive fusion strategy outperform fixed-weight or concatenative fusion methods in terms of accuracy and robustness?
• RQ5 Can the integration of saliency features be done with minimal computational overhead to preserve real-time performance?

Key findings

• ECOSAL achieves a mean accuracy of 0.715 on the OTB-2013 dataset under one-pass evaluation, outperforming state-of-the-art methods including ECO*, MDNet, SANet, MCPF, and CCOT.
• On the UAV-123 benchmark, ECOSAL achieves superior performance in 10 out of 12 challenging cases, with the highest mean accuracy of 0.707 across all sequences.
• The tracker shows consistent improvement over the baseline ECO in all challenging cases except full occlusion (FO) and low resolution (LR), where performance is slightly worse due to limited spatial support.
• Robustness analysis confirms that ECOSAL significantly outperforms ECO in both spatial (e.g., illumination variance, occlusion) and temporal (e.g., in-plane rotation, deformation) challenges.
• The method operates at 9.3 fps, introducing only a minor computational overhead compared to the baseline ECO, which runs at 11 fps, demonstrating effective real-time efficiency.
• The adaptive weighting mechanism successfully suppresses temporally inconsistent saliency predictions, enhancing tracker stability during appearance and viewpoint variations.