[Paper Review] Multi-target multi-camera vehicle tracking using transformer-based camera link model and spatial-temporal information

Hsiang-Wei Huang, Yang, Cheng-Yen+1 more|arXiv (Cornell University)|Jan 18, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a transformer-based camera link model that integrates spatial-temporal filtering to improve multi-target multi-camera vehicle tracking (MTMCT). By leveraging temporal and directional masks and bi-directional transition time windows, the method significantly reduces the search space for cross-camera association, achieving an IDF1 score of 73.68% on the CityFlowV2 benchmark—outperforming all teams in the 2020 AI City Challenge and ranking among the top in 2021.

ABSTRACT

Multi-target multi-camera tracking (MTMCT) of vehicles, i.e. tracking vehicles across multiple cameras, is a crucial application for the development of smart city and intelligent traffic system. The main challenges of MTMCT of vehicles include the intra-class variability of the same vehicle and inter-class similarity between different vehicles and how to associate the same vehicle accurately across different cameras under large search space. Previous methods for MTMCT usually use hierarchical clustering of trajectories to conduct cross camera association. However, the search space can be large and does not take spatial and temporal information into consideration. In this paper, we proposed a transformer-based camera link model with spatial and temporal filtering to conduct cross camera tracking. Achieving 73.68% IDF1 on the Nvidia Cityflow V2 dataset test set, showing the effectiveness of our camera link model on multi-target multi-camera tracking.

Motivation & Objective

• To address the challenges of large search space, intra-class variability, and inter-class similarity in multi-target multi-camera vehicle tracking (MTMCT).
• To improve cross-camera association accuracy by incorporating spatial and temporal constraints into the camera link model.
• To reduce false positives in Re-ID by filtering candidate matches using motion direction and transition time windows.
• To leverage transformer-based feature extraction for more discriminative appearance features in vehicle re-identification.
• To outperform existing hierarchical clustering-based methods in MTMCT by integrating spatial-temporal priors into a unified tracking framework.

Proposed method

• A transformer-based Re-ID model is used to extract discriminative appearance features, leveraging self-attention to capture long-range dependencies without downsampling.
• Temporal and direction masks are applied to filter out unlikely cross-camera associations based on vehicle movement direction and temporal consistency.
• Bi-directional transition time windows (BTT) are introduced to constrain the time window in which a vehicle can appear in adjacent cameras, reducing candidate matches.
• The camera link model uses cosine distance between appearance features to associate trajectories, with thresholds set at 0.45 for similarity and trajectory length filtering at 2–2000 frames.
• The system integrates single-camera tracking (SCT) with SORT-like association using appearance and motion cues, followed by transformer-enhanced Re-ID and spatial-temporal filtering in the camera link model.
• Data augmentation techniques including color normalization, random crop, and random flip are applied during Re-ID training to improve robustness.

Figure 1: The framework of the transformer-based ReID model from [ 3 ] . The BNNeck is introduced inspired by [ 11 ] .

Experimental results

Research questions

• RQ1 Can a transformer-based Re-ID model improve appearance feature discrimination in multi-camera vehicle tracking compared to CNN-based models?
• RQ2 How does incorporating spatial-temporal filtering reduce the search space for cross-camera association in MTMCT?
• RQ3 To what extent do temporal and directional masks improve IDF1 in multi-camera vehicle tracking?
• RQ4 Can bi-directional transition time windows further enhance association accuracy by modeling vehicle movement across camera networks?
• RQ5 How does the proposed camera link model compare to hierarchical clustering baselines in terms of IDF1 and robustness on real-world datasets?

Key findings

• The proposed method achieves an IDF1 score of 73.68% on the CityFlowV2 test set, significantly outperforming the baseline method (34.74% IDF1).
• The addition of temporal and direction masks (TDM) improves IDF1 by 22.35 percentage points, demonstrating the effectiveness of spatial-temporal filtering.
• The integration of bi-directional transition time windows (BTT) further boosts IDF1 by 26.59 percentage points over the baseline, achieving state-of-the-art performance.
• The system outperforms all teams in the 2020 AI City Challenge on the City-scale MTMC vehicle tracking track, with an IDF1 of 73.68% compared to the second-place team’s 45.85%.
• In the 2021 AI City Challenge, the method ranked fourth overall, with an IDF1 of 73.68%, behind only three teams that achieved higher scores.
• The ablation study confirms that spatial-temporal filtering significantly reduces false positive associations, leading to improved ID precision (67.73%) and ID recall (80.77%) in the final system.

Figure 2: Vehicles detected entering the intersection while the traffic light is red will be matched with trajectories in the current camera. On the other hand, vehicles leaving the camera’s field of view through the green traffic zone will be matched with trajectories in the next adjacent camera.