[Paper Review] Learning to Update for Object Tracking.

Li Bi, Wenxuan Xie+2 more|arXiv (Cornell University)|Jun 19, 2018

Video Surveillance and Tracking Methods26 references3 citations

TL;DR

This paper proposes 'learning to update'—a method that trains a recurrent neural network (RNN) to learn the online learning algorithm itself from large-scale offline video data, enabling adaptive model updates in object trackers. The learned updater improves both template-based and correlation filter-based trackers, achieving state-of-the-art performance among real-time GPU trackers while running faster than real time with low memory usage.

ABSTRACT

Model update lies at the heart of object tracking.Generally, model update is formulated as an online learning problem where a target model is learned over the online training dataset. Our key innovation is to \emph{learn the online learning algorithm itself using large number of offline videos}, i.e., \emph{learning to update}. The learned updater takes as input the online training dataset and outputs an updated target model. As a first attempt, we design the learned updater based on recurrent neural networks (RNNs) and demonstrate its application in a template-based tracker and a correlation filter-based tracker. Our learned updater consistently improves the base trackers and runs faster than realtime on GPU while requiring small memory footprint during testing. Experiments on standard benchmarks demonstrate that our learned updater outperforms commonly used update baselines including the efficient exponential moving average (EMA)-based update and the well-designed stochastic gradient descent (SGD)-based update. Equipped with our learned updater, the template-based tracker achieves state-of-the-art performance among realtime trackers on GPU.

Motivation & Objective

• To address the limitation of hand-designed online update rules in object tracking by learning the update process itself.
• To improve tracker robustness and accuracy by learning from diverse offline video sequences rather than relying on fixed update heuristics.
• To develop a lightweight, fast, and memory-efficient update mechanism suitable for real-time inference on GPU.
• To demonstrate generalization of the learned updater across different tracker architectures, including template-based and correlation filter-based trackers.

Proposed method

• Train a recurrent neural network (RNN) to learn the online learning algorithm by supervising it on a large-scale offline video dataset.
• Use the RNN-based updater to process the online training sequence and generate an updated target model, replacing traditional update rules.
• Integrate the learned updater into both template-based and correlation filter-based trackers as a plug-in module.
• Train the updater end-to-end using a large number of video sequences, optimizing for tracking accuracy and update stability.
• Design the RNN to take feature representations of online samples and output a refined model update vector.
• Ensure the updater is efficient at inference by minimizing parameters and computational complexity, enabling real-time performance on GPU.

Experimental results

Research questions

• RQ1 Can a neural network be trained to learn the online update process in object tracking more effectively than hand-designed rules?
• RQ2 Does the learned updater generalize across different tracker architectures such as template-based and correlation filter trackers?
• RQ3 Can the learned updater achieve real-time performance with low memory usage while improving tracking accuracy?
• RQ4 How does the learned updater compare to standard baselines like EMA and SGD in terms of tracking performance and robustness?

Key findings

• The learned updater consistently improves the performance of both template-based and correlation filter-based trackers on standard benchmarks.
• The method achieves state-of-the-art accuracy among real-time trackers on GPU when integrated into a template-based tracker.
• The learned updater runs faster than real time on GPU, with a small memory footprint during inference.
• The RNN-based updater outperforms both exponential moving average (EMA)-based and stochastic gradient descent (SGD)-based update baselines in tracking accuracy.
• The learned updater generalizes well across different tracking architectures, demonstrating transferability and robustness.