[Paper Review] A multi-feature tracking algorithm enabling adaptation to context variations

Duc Phu Chau, François Brémond+1 more|arXiv (Cornell University)|Dec 6, 2011

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a multi-feature tracking algorithm that adapts to varying scene contexts by learning optimal feature weights via Adaboost. It combines 8 features—including HOG, color histograms, and spatial pyramid distances—using a temporal window and trajectory filtering to improve robustness, especially under occlusion and illumination changes, outperforming state-of-the-art methods in public datasets and real-world video sequences.

ABSTRACT

We propose in this paper a tracking algorithm which is able to adapt itself to different scene contexts. A feature pool is used to compute the matching score between two detected objects. This feature pool includes 2D, 3D displacement distances, 2D sizes, color histogram, histogram of oriented gradient (HOG), color covariance and dominant color. An offline learning process is proposed to search for useful features and to estimate their weights for each context. In the online tracking process, a temporal window is defined to establish the links between the detected objects. This enables to find the object trajectories even if the objects are misdetected in some frames. A trajectory filter is proposed to remove noisy trajectories. Experimentation on different contexts is shown. The proposed tracker has been tested in videos belonging to three public datasets and to the Caretaker European project. The experimental results prove the effect of the proposed feature weight learning, and the robustness of the proposed tracker compared to some methods in the state of the art. The contributions of our approach over the state of the art trackers are: (i) a robust tracking algorithm based on a feature pool, (ii) a supervised learning scheme to learn feature weights for each context, (iii) a new method to quantify the reliability of HOG descriptor, (iv) a combination of color covariance and dominant color features with spatial pyramid distance to manage the case of object occlusion.

Motivation & Objective

• To develop a tracking algorithm that automatically adapts to diverse scene contexts such as object density, illumination, and occlusion frequency.
• To address the challenge of manually tuning feature weights for different environments by introducing an offline learning process.
• To improve tracking robustness in complex scenarios like low contrast, depth variation, and object occlusion through feature fusion and reliability quantification.
• To reduce false positives and trajectory loss by combining temporal consistency with a trajectory filtering mechanism.
• To enable context-aware tracking without requiring prior knowledge of object paths or fixed motion models.

Proposed method

• Constructs a feature pool of eight descriptors: 2D/3D displacement distance, 2D size ratio and area, color histogram, HOG, color covariance, and dominant color.
• Computes link similarity between detected objects using weighted combinations of these features within a temporal window to establish object correspondence.
• Employs an offline Adaboost-based learning scheme to automatically determine optimal feature weights for each specific context using labeled training sequences.
• Introduces a reliability metric for the HOG descriptor to improve its contribution in feature weighting, especially under poor visibility.
• Fuses color covariance and dominant color with spatial pyramid distance to enhance performance during partial or full occlusion.
• Applies a trajectory filter that removes noisy tracks by evaluating consistency and temporal coherence across frames.

Experimental results

Research questions

• RQ1 How can a tracking system automatically adapt its feature selection and weighting to different scene contexts without manual tuning?
• RQ2 What combination of visual features provides the most robust tracking performance under challenging conditions such as occlusion and low illumination?
• RQ3 Can a supervised learning approach like Adaboost effectively learn context-specific feature weights to improve tracking accuracy and stability?
• RQ4 How does the integration of spatial pyramid distance with color features improve tracking robustness during object occlusion?
• RQ5 To what extent does the proposed method outperform existing state-of-the-art trackers in diverse real-world video sequences?

Key findings

• The proposed tracker achieved the highest performance in 12 out of 16 evaluation metrics across two ETISEO video sequences (BE and MO), with the highest average F1-score of 0.93 for the BE sequence.
• In the Caviar dataset, the tracker successfully tracked all 9 trajectories (TP=9) with no false negatives (FN=0), while reducing false positives from 7 (in [snidaro]) to 3.
• For the low-contrast, high-depth sequence (ETI-VS1-MO-7-C1), the learning process selected color covariance as the dominant feature, confirming its effectiveness under poor visibility.
• The tracker demonstrated superior robustness in the single-person sequence (ETI-VS1-BE-18-C4), where 3D displacement distance was identified as the most discriminative feature.
• The combination of color covariance, dominant color, and spatial pyramid distance significantly improved occlusion handling compared to individual features.
• The offline Adaboost learning process successfully identified context-specific feature weights, validating the method's adaptability across diverse environments.