[Paper Review] Online Adaptive Hidden Markov Model for Multi-Tracker Fusion

Vojir, Tomas, Jiřı́ Matas+1 more|arXiv (Cornell University)|Apr 23, 2015

Video Surveillance and Tracking Methods22 references4 citations

TL;DR

This paper proposes HMMTxD, an online adaptive hidden Markov model that fuses multiple complementary trackers and a detector for robust visual object tracking. By modeling tracker failure states as latent HMM states and using detector outputs to supervise unsupervised Baum-Welch learning, HMMTxD achieves state-of-the-art performance across multiple benchmarks, significantly outperforming existing fusion methods in accuracy and robustness to occlusions.

ABSTRACT

In this paper, we propose a novel method for visual object tracking called HMMTxD. The method fuses observations from complementary out-of-the box trackers and a detector by utilizing a hidden Markov model whose latent states correspond to a binary vector expressing the failure of individual trackers. The Markov model is trained in an unsupervised way, relying on an online learned detector to provide a source of tracker-independent information for a modified Baum- Welch algorithm that updates the model w.r.t. the partially annotated data. We show the effectiveness of the proposed method on combination of two and three tracking algorithms. The performance of HMMTxD is evaluated on two standard benchmarks (CVPR2013 and VOT) and on a rich collection of 77 publicly available sequences. The HMMTxD outperforms the state-of-the-art, often significantly, on all datasets in almost all criteria.

Motivation & Objective

• To address the limitations of individual trackers that fail under domain shifts or complex motions.
• To enable robust multi-tracker fusion by modeling tracker reliability and failure states using a hidden Markov model.
• To develop an online, unsupervised learning framework that leverages detector outputs for partial supervision without requiring full annotation.
• To improve long-term tracking robustness by enabling dynamic reinitialization of failed trackers based on detector confidence.
• To fuse diverse trackers with non-uniform confidence estimates by learning observable distributions in a probabilistic HMM framework.

Proposed method

• The HMM models tracker states as binary vectors indicating failure or success of each tracker, with latent states representing combinations of tracker performance.
• A modified Baum-Welch algorithm performs unsupervised HMM training using partially annotated data from an online-learned object detector.
• Detector outputs serve as ground truth for model updates only when consistent with the current most probable HMM state, reducing error propagation.
• The method uses a generalized method of moments to estimate beta-distributed observable densities for tracker confidence scores, enabling probabilistic fusion.
• Transition probabilities are updated based on expected state transitions, with detector-verified frames used to constrain learning and improve long-term stability.
• The final pose estimate combines tracker outputs weighted by their posterior probabilities, integrating past performance and current observations.

Experimental results

Research questions

• RQ1 Can an online, unsupervised HMM effectively fuse multiple complementary trackers with non-uniform confidence metrics?
• RQ2 How can a high-precision detector be used to supervise HMM training without full annotation?
• RQ3 Does the proposed method improve long-term tracking robustness after occlusions or tracker failures?
• RQ4 Can the HMM framework outperform rule-based or fixed-weight fusion strategies on diverse benchmarks?
• RQ5 To what extent does detector consistency with the HMM state improve model learning and tracking accuracy?

Key findings

• HMMTxD outperforms state-of-the-art methods on the CVPR2013 benchmark, achieving higher precision and success rates across all evaluation criteria.
• On the VOT benchmark, HMMTxD significantly improves tracking accuracy and robustness, particularly in handling occlusions and long-term tracking challenges.
• In experiments on 77 publicly available video sequences, HMMTxD demonstrated consistent superiority over baseline fusion methods and individual trackers.
• The method achieves improved long-term tracking performance due to detector-supervised reinitialization of failed trackers, reducing drift and failure propagation.
• The use of detector outputs as partial supervision enables effective online learning, with the HMM adapting to changing tracker reliability over time.
• The generalized method of moments for beta-distributed confidence scores enables accurate modeling of tracker observables without strong parametric assumptions.