[Paper Review] State-aware Re-identification Feature for Multi-target Multi-camera Tracking

Peng Li, Jiabin Zhang+4 more|arXiv (Cornell University)|Jun 4, 2019

Video Surveillance and Tracking Methods65 references4 citations

TL;DR

This paper proposes a state-aware re-identification (Re-ID) feature for multi-target multi-camera tracking (MTMCT) that leverages human pose information to estimate occlusion status and orientation, enhancing appearance feature robustness. By fusing cluster-based, orientation-aware, and temporal validity features into a robust tracking feature, and using tracklet association with a redesigned distance matrix, the method achieves 81.3% IDF1 on the DukeMTMCT hard sequence, outperforming all prior methods.

ABSTRACT

Multi-target Multi-camera Tracking (MTMCT) aims to extract the trajectories from videos captured by a set of cameras. Recently, the tracking performance of MTMCT is significantly enhanced with the employment of re-identification (Re-ID) model. However, the appearance feature usually becomes unreliable due to the occlusion and orientation variance of the targets. Directly applying Re-ID model in MTMCT will encounter the problem of identity switches (IDS) and tracklet fragment caused by occlusion. To solve these problems, we propose a novel tracking framework in this paper. In this framework, the occlusion status and orientation information are utilized in Re-ID model with human pose information considered. In addition, the tracklet association using the proposed fused tracking feature is adopted to handle the fragment problem. The proposed tracker achieves 81.3\% IDF1 on the multiple-camera hard sequence, which outperforms all other reference methods by a large margin.

Motivation & Objective

• To address identity switches and tracklet fragmentation in multi-target multi-camera tracking (MTMCT) caused by occlusion and appearance variance.
• To improve Re-ID feature reliability in crowded scenes where detectors produce low-quality detections.
• To enhance trajectory consistency by incorporating occlusion status and orientation cues into the Re-ID feature learning process.
• To design a robust fused tracking feature that combines cluster-based, orientation-aware, and temporal validity features for stable data association.
• To achieve state-of-the-art performance on the DukeMTMCT benchmark, especially on the challenging hard sequence.

Proposed method

• Uses human pose estimation to infer target state, including occlusion status and orientation, to guide Re-ID feature extraction.
• Introduces a fused tracking feature that combines current valid Re-ID features, cluster-based features from tracklets, and orientation-aware features.
• Applies a temporal validity mask ($f_{invalid}$) to suppress unreliable features from highly occluded frames.
• Designs a novel distance matrix for data association that incorporates state-aware features to improve robustness under occlusion.
• Employs an offline tracklet association strategy: first generating short tracklets, then rectifying and clustering them to resolve fragmentation.
• Uses Hungarian algorithm for global trajectory matching based on the fused feature distance matrix.

Experimental results

Research questions

• RQ1 How can occlusion status and orientation be effectively modeled to improve Re-ID feature reliability in multi-camera tracking?
• RQ2 To what extent does fusing cluster-based, orientation-aware, and temporal validity features improve tracking performance?
• RQ3 Can a redesigned distance matrix based on state-aware features reduce identity switches and trajectory fragmentation?
• RQ4 How does the proposed tracklet association strategy compare to frame-by-frame online association in handling occlusion?
• RQ5 What is the contribution of each component (pose-based state, cluster, orientation, invalid mask) to the final tracking performance?

Key findings

• The proposed tracker achieves 81.3% IDF1 on the DukeMTMCT hard sequence, setting a new state-of-the-art performance.
• The ablation study shows that adding cluster-based features improves IDF1 by 5.4% over the baseline.
• Incorporating orientation-aware features further increases IDF1 by 2.6% compared to the cluster-only baseline.
• Adding the temporal invalid feature reduces identity switches (IDS) from 6,564 to 5,466 while improving IDF1 by 0.1%.
• The full model with all components achieves 85.2% IDF1 on the trainval-mini set, demonstrating the effectiveness of the state-aware Re-ID feature.
• The method outperforms all published and unpublished methods on the benchmark, especially on the hard sequence, confirming robustness in crowded, occlusion-heavy scenes.