[Paper Review] Strong-TransCenter: Improved Multi-Object Tracking based on Transformers with Dense Representations

Amit Galor, Roy Orfaig+1 more|arXiv (Cornell University)|Oct 24, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes Strong-TransCenter, an improved transformer-based multi-object tracking framework that enhances TransCenter with post-processing using Kalman filtering for motion modeling and a deep embedding network for re-identification. The method achieves state-of-the-art performance on MOT17 and MOT20 benchmarks, ranking first among transformer-based trackers in HOTA and IDF1 metrics with 70.9% IDF1, 59.8% HOTA, and 75.8% MOTA on MOT17, and 67.5% IDF1, 56.3% HOTA, and 73.0% MOTA on MOT20.

ABSTRACT

Transformer networks have been a focus of research in many fields in recent years, being able to surpass the state-of-the-art performance in different computer vision tasks. However, in the task of Multiple Object Tracking (MOT), leveraging the power of Transformers remains relatively unexplored. Among the pioneering efforts in this domain, TransCenter, a Transformer-based MOT architecture with dense object queries, demonstrated exceptional tracking capabilities while maintaining reasonable runtime. Nonetheless, one critical aspect in MOT, track displacement estimation, presents room for enhancement to further reduce association errors. In response to this challenge, our paper introduces a novel improvement to TransCenter. We propose a post-processing mechanism grounded in the Track-by-Detection paradigm, aiming to refine the track displacement estimation. Our approach involves the integration of a carefully designed Kalman filter, which incorporates Transformer outputs into measurement error estimation, and the use of an embedding network for target re-identification. This combined strategy yields substantial improvement in the accuracy and robustness of the tracking process. We validate our contributions through comprehensive experiments on the MOTChallenge datasets MOT17 and MOT20, where our proposed approach outperforms other Transformer-based trackers. The code is publicly available at: https://github.com/amitgalor18/STC_Tracker

Motivation & Objective

• To improve the performance of transformer-based multi-object tracking by integrating post-processing components into the TransCenter framework.
• To reduce ID switches and misdetections in challenging tracking scenarios such as occlusions and motion blur.
• To enhance tracking robustness through motion modeling and appearance-based re-identification without modifying the backbone transformer architecture.
• To achieve state-of-the-art results on the MOTChallenge benchmark, particularly in HOTA and IDF1, which are more sensitive to tracking accuracy and consistency.

Proposed method

• The tracker uses a pre-trained TransCenter backbone that generates dense object queries for center-based tracking across feature maps.
• Post-processing includes a Kalman filter to estimate motion trajectories and improve prediction accuracy between frames.
• Appearance-based re-identification is performed using a learned embedding network to match detections to existing tracks based on visual similarity.
• Track association combines IoU-based matching with appearance embedding distances, using the minimum of the two after filtering with a high IoU threshold to ensure spatial consistency.
• The system leverages the Hungarian algorithm for optimal assignment between detections and tracks, integrating motion and appearance cues.
• The framework is trained end-to-end on detection and tracking heads, with post-processing applied during inference only.

Experimental results

Research questions

• RQ1 Can post-processing with Kalman filtering and appearance re-identification significantly improve the performance of a state-of-the-art transformer-based MOT tracker?
• RQ2 How does the integration of motion modeling and appearance features affect ID switch and false positive rates in occluded or ambiguous tracking scenarios?
• RQ3 To what extent do HOTA and IDF1 metrics improve when combining motion prediction and embedding-based re-identification in a transformer-based tracker?
• RQ4 Does the proposed method outperform existing transformer-based trackers on standard MOT benchmarks like MOT17 and MOT20?

Key findings

• Strong-TransCenter achieves 70.9% IDF1, 59.8% HOTA, and 75.8% MOTA on the MOT17 test set, ranking first among transformer-based trackers in IDF1 and HOTA.
• On the MOT20 test set, the tracker achieves 67.5% IDF1, 56.3% HOTA, and 73.0% MOTA, setting new SOTA results in all three metrics among transformer-based trackers.
• Qualitative analysis shows a significant reduction in ID switch errors, particularly in scenarios involving object emergence or occlusion, due to improved motion prediction from the Kalman filter.
• The Kalman filter enables more accurate trajectory prediction, reducing the reliance on noisy optical flow and improving consistency in track continuity.
• The combination of appearance embedding and position-based matching reduces false positives and false negatives, especially in partially occluded cases, as demonstrated in frame-level visualizations.
• The tracker maintains strong performance on MOTA while significantly improving HOTA and IDF1, indicating better overall tracking quality and identity consistency.