[Paper Review] Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning

Ilchae Jung, Kihyun You+3 more|arXiv (Cornell University)|Nov 25, 2019

Video Surveillance and Tracking Methods35 references4 citations

TL;DR

This paper proposes MetaRTT, a meta-learning framework for real-time object tracking that enables fast model adaptation and one-shot channel pruning using only the first-frame ground truth. By simulating tracking episodes with hard examples and meta-optimizing initial weights, learning rates, and pruning masks, MetaRTT achieves state-of-the-art accuracy and 38% faster inference than RT-MDNet, with a 12% speedup via pruning while maintaining near-identical performance.

ABSTRACT

We propose a novel meta-learning framework for real-time object tracking with efficient model adaptation and channel pruning. Given an object tracker, our framework learns to fine-tune its model parameters in only a few iterations of gradient-descent during tracking while pruning its network channels using the target ground-truth at the first frame. Such a learning problem is formulated as a meta-learning task, where a meta-tracker is trained by updating its meta-parameters for initial weights, learning rates, and pruning masks through carefully designed tracking simulations. The integrated meta-tracker greatly improves tracking performance by accelerating the convergence of online learning and reducing the cost of feature computation. Experimental evaluation on the standard datasets demonstrates its outstanding accuracy and speed compared to the state-of-the-art methods.

Motivation & Objective

• To address the challenge of slow online model adaptation in deep learning-based object trackers, especially under real-time constraints.
• To enable efficient, target-specific model compression (channel pruning) using only a single ground-truth annotation at the first frame.
• To develop a meta-learning framework that simulates both initial and online adaptation phases for improved generalization and convergence speed.
• To reduce computational cost without sacrificing tracking accuracy by jointly optimizing adaptation and pruning through meta-training.
• To demonstrate that meta-learned hyperparameters generalize well across benchmarks without manual tuning.

Proposed method

• The framework formulates object tracking as a meta-learning problem, where meta-parameters (initial weights, learning rates, pruning masks) are optimized over simulated tracking episodes.
• It simulates two adaptation phases: (1) initial adaptation using the first-frame ground truth, and (2) online adaptation using tracked targets from previous frames.
• Hard examples are incorporated during meta-training to improve robustness to challenging tracking scenarios.
• A one-shot channel pruning technique is developed using meta-learned pruning masks, which are optimized based on a single ground-truth annotation.
• The meta-tracker is trained via end-to-end optimization over diverse simulated tracking episodes, enabling fast convergence during inference.
• Pruning is applied to network channels using a learned mask, reducing FLOPs and inference time with minimal accuracy drop.

Experimental results

Research questions

• RQ1 Can meta-learning be effectively applied to real-time object tracking to accelerate online model adaptation?
• RQ2 Can one-shot channel pruning be learned via meta-learning using only the first-frame ground truth, without iterative fine-tuning?
• RQ3 Does simulating both initial and online adaptation phases in meta-training lead to better generalization than prior meta-learning approaches?
• RQ4 Can meta-learned hyperparameters generalize across datasets without re-tuning?
• RQ5 To what extent does joint optimization of adaptation and pruning improve speed-accuracy trade-offs in real-time tracking?

Key findings

• MetaRTT achieves a 38% speedup over RT-MDNet (58 FPS vs. 42 FPS) while improving success rate from 65.0% to 65.5% on OTB2015.
• MetaRTT+Prune achieves 65 FPS with only a 0.4% drop in success rate (65.1% vs. 65.5%) and 12% faster inference than MetaRTT.
• One-shot pruning with meta-learning reduces model parameters by 50±5% with minimal accuracy loss, outperforming non-meta and static pruning baselines.
• MetaRTT+Prune maintains competitive performance on TempleColor and OTB2015, outperforming state-of-the-art real-time trackers in accuracy and speed.
• Meta-learned hyperparameters generalize well: no manual tuning was needed for OTB2015, TempleColor, VOT2016, or UAV123.
• MetaRTT+COCO further improves performance on VOT2016 (EAO: 0.350) and UAV123 (Succ: 56.6%, Prec: 80.9%), suggesting potential for further gains with larger pretraining data.