[Paper Review] Learning Spatial-Temporal Regularized Correlation Filters for Visual Tracking

Feng Li, Tian Cheng+3 more|arXiv (Cornell University)|Mar 23, 2018

Video Surveillance and Tracking MethodsComputer Science2 references70 citations

TL;DR

STRCF introduces temporal regularization to SRDCF and solves with ADMM, achieving real-time tracking speed and improved accuracy over SRDCF on multiple benchmarks.

ABSTRACT

Discriminative Correlation Filters (DCF) are efficient in visual tracking but suffer from unwanted boundary effects. Spatially Regularized DCF (SRDCF) has been suggested to resolve this issue by enforcing spatial penalty on DCF coefficients, which, inevitably, improves the tracking performance at the price of increasing complexity. To tackle online updating, SRDCF formulates its model on multiple training images, further adding difficulties in improving efficiency. In this work, by introducing temporal regularization to SRDCF with single sample, we present our spatial-temporal regularized correlation filters (STRCF). Motivated by online Passive-Agressive (PA) algorithm, we introduce the temporal regularization to SRDCF with single sample, thus resulting in our spatial-temporal regularized correlation filters (STRCF). The STRCF formulation can not only serve as a reasonable approximation to SRDCF with multiple training samples, but also provide a more robust appearance model than SRDCF in the case of large appearance variations. Besides, it can be efficiently solved via the alternating direction method of multipliers (ADMM). By incorporating both temporal and spatial regularization, our STRCF can handle boundary effects without much loss in efficiency and achieve superior performance over SRDCF in terms of accuracy and speed. Experiments are conducted on three benchmark datasets: OTB-2015, Temple-Color, and VOT-2016. Compared with SRDCF, STRCF with hand-crafted features provides a 5 times speedup and achieves a gain of 5.4% and 3.6% AUC score on OTB-2015 and Temple-Color, respectively. Moreover, STRCF combined with CNN features also performs favorably against state-of-the-art CNN-based trackers and achieves an AUC score of 68.3% on OTB-2015.

Motivation & Objective

• Address boundary effects in discriminative correlation filters (DCFs) for visual tracking.
• Propose a spatial-temporal regularized DCF (STRCF) that updates from a single frame with temporal regularization.
• Develop an efficient ADMM-based solver with closed-form subproblems.
• Show STRCF provides robust appearance models under large appearance variations while maintaining real-time speed.

Proposed method

• Introduce a temporal regularization term mu/2 * ||f - f_{t-1}||^2 to SRDCF, forming STRCF (Eq. 2).
• Solve the convex STRCF objective via ADMM with an auxiliary variable g and alternating updates.
• In the f-subproblem, solve in the Fourier domain per pixel location using Parseval’s theorem and Sherman–Morrison formula for efficiency (Eq. 9–12).
• In the g-subproblem, obtain a closed-form solution leveraging the diagonal structure (Eq. 13).
• Update the ADMM penalty parameter gamma iteratively (Eq. 14).
• Analyze computational complexity as O(DMN log(MN)) per frame, with overall cost O(DMN log(MN) NI).
• Establish convergence guarantees (convex problem; Eckstein–Bertsekas conditions) and empirical two-iteration convergence.

Experimental results

Research questions

• RQ1 Can STRCF approximate the SRDCF model learned on multiple training images while maintaining higher efficiency?
• RQ2 Does incorporating temporal regularization improve robustness to appearance variations and occlusions compared with SRDCF?
• RQ3 What is the impact of the temporal regularization parameter mu on tracking performance?
• RQ4 Can STRCF achieve real-time performance with hand-crafted and deep features while maintaining competitive accuracy?

Key findings

• STRCF achieves about a 5.7% mean OP gain over SRDCF on OTB-2015 and Temple-Color.
• STRCF runs in real time (approximately 30 FPS) with hand-crafted features, with 31.5 FPS for STRCF(HOG) and 24.3 FPS for STRCF(HOGCN).
• STRCF with temporal regularization provides robust updates, delivering up to 14.5% and 5.7% gains on the OV and OCC attributes, respectively, over SRDCF variants.
• DeepSTRCF (STRCF with CNN features) attains a mean OP of 84.2% on OTB-2015, outperforming DeepSRDCF by 7.4%.
• On VOT-2016, STRCF achieves an EAO of 0.279 (STRCF) and 0.313 (DeepSTRCF), with DeepSTRCF showing higher EAO among the CNN-augmented variants.
• On Temple-Color, STRCF is competitive with ECO-HC and DeepSTRCF yields the best performance among reported results on that dataset.