[Paper Review] TransTrack: Multiple Object Tracking with Transformer

Peize Sun, Jinkun Cao+6 more|arXiv (Cornell University)|Dec 31, 2020

Video Surveillance and Tracking Methods62 references359 citations

TL;DR

TransTrack introduces a transformer-based joint-detection-and-tracking framework that uses two query sources—learned object queries for detection and track queries from previous frames for propagation—to perform MOT in a single shot. It achieves competitive MOTA on MOT17 and MOT20 without a separate Re-ID module or NMS in detection.

ABSTRACT

In this work, we propose TransTrack, a simple but efficient scheme to solve the multiple object tracking problems. TransTrack leverages the transformer architecture, which is an attention-based query-key mechanism. It applies object features from the previous frame as a query of the current frame and introduces a set of learned object queries to enable detecting new-coming objects. It builds up a novel joint-detection-and-tracking paradigm by accomplishing object detection and object association in a single shot, simplifying complicated multi-step settings in tracking-by-detection methods. On MOT17 and MOT20 benchmark, TransTrack achieves 74.5\% and 64.5\% MOTA, respectively, competitive to the state-of-the-art methods. We expect TransTrack to provide a novel perspective for multiple object tracking. The code is available at: \url{https://github.com/PeizeSun/TransTrack}.

Motivation & Objective

• Motivate a simple, efficient MOT approach that unifies detection and association in a single stage.
• Leverage transformer attention to propagate previously detected objects while detecting new ones.
• Eliminate or reduce multi-stage pipelines and post-processing like NMS in detection.
• Demonstrate competitive MOT performance on MOT17 and MOT20 benchmarks.
• Provide insights into design choices for query inputs and matching strategies in MOT.

Proposed method

• Use a two-branch decoder architecture where one decoder predicts detection boxes from learned object queries, and the other predicts tracking boxes from track queries derived from previous frame objects.
• Employ a shared encoder on fused features from two consecutive frames as the backbone for both decoders.
• Output two sets of boxes (detection and tracking) and use IoU-based Hungarian matching to form the final track set.
• Train with set-based loss (classification, L1, and generalized IoU) over matched pairs, following DETR-style loss.
• Adopt deformable transformer as the default architecture to fuse multi-scale features and improve accuracy.
• During inference, perform track rebirth for robustness against occlusion by keeping unmatched tracking boxes active for K frames.

Experimental results

Research questions

• RQ1 Can a transformer-based MOT model jointly detect and associate objects across frames without a separate Re-ID module?
• RQ2 How does incorporating track queries from previous frames affect object association and detection robustness in MOT?
• RQ3 What is the impact of using dual query streams (object and track queries) on MOT performance metrics like MOTA and IDs?
• RQ4 What architectural choices (e.g., deformable transformer) maximize MOT performance while maintaining reasonable speed?
• RQ5 How does TransTrack compare to detector-plus-motion or detector-plus-Re-ID baselines on crowded MOT benchmarks?

Key findings

• TransTrack achieves 74.5 MOTA on MOT17 and 64.5 MOTA on MOT20 under private detector settings, demonstrating competitive performance.
• Using both object queries and track queries significantly improves detection and tracking over using either alone.
• Deformable Transformer as the backbone yields the best MOTA (65.0) among tested architectures, with strong localization (MOTP) and low IDs.
• Two-decoder design with IoU-based Hungarian matching can replace traditional NMS and provide robust association across frames.
• Incorporating CrowdHuman as pre-training data plus MOT fine-tuning boosts MOT17 performance by a notable margin (pre-training on CrowdHuman improves MOTA to 64.8 from 61.6).
• Motion-model comparison shows TransTrack’s track-query approach provides comparable or better ID-switch robustness than Kalman-filter-based methods, especially when sampling fewer frames.