[Paper Review] TransCenter: Transformers with Dense Queries for Multiple-Object Tracking

Yihong Xu, Yutong Ban+4 more|arXiv (Cornell University)|Jul 22, 2021

Video Surveillance and Tracking Methods69 references75 citations

TL;DR

TransCenter introduces a center-based transformer MOT framework using image-related dense detection queries and sparse tracking queries, achieving state-of-the-art results on MOT benchmarks.

ABSTRACT

Transformers have proven superior performance for a wide variety of tasks since they were introduced. In recent years, they have drawn attention from the vision community in tasks such as image classification and object detection. Despite this wave, an accurate and efficient multiple-object tracking (MOT) method based on transformers is yet to be designed. We argue that the direct application of a transformer architecture with quadratic complexity and insufficient noise-initialized sparse queries - is not optimal for MOT. We propose TransCenter, a transformer-based MOT architecture with dense representations for accurately tracking all the objects while keeping a reasonable runtime. Methodologically, we propose the use of image-related dense detection queries and efficient sparse tracking queries produced by our carefully designed query learning networks (QLN). On one hand, the dense image-related detection queries allow us to infer targets' locations globally and robustly through dense heatmap outputs. On the other hand, the set of sparse tracking queries efficiently interacts with image features in our TransCenter Decoder to associate object positions through time. As a result, TransCenter exhibits remarkable performance improvements and outperforms by a large margin the current state-of-the-art methods in two standard MOT benchmarks with two tracking settings (public/private). TransCenter is also proven efficient and accurate by an extensive ablation study and comparisons to more naive alternatives and concurrent works. For scientific interest, the code is made publicly available at https://github.com/yihongxu/transcenter.

Motivation & Objective

• Motivate and design a transformer-based MOT method that avoids gaps from sparse queries and over-detections in crowded scenes.
• Introduce image-related dense detection queries to provide global, robust detections.
• Develop sparse tracking queries and a specialized decoder to efficiently associate objects across frames.
• Reduce computational complexity to achieve efficient MOT with dense queries.
• Provide variants (TransCenter, TransCenter-Dual, TransCenter-Lite) to balance accuracy and efficiency.

Proposed method

• Use a weight-shared transformer encoder (PVT-based) to produce multi-scale dense memories from consecutive frames.
• Introduce Query Learning Networks (QLN) to generate dense detection queries and sparse tracking queries from encoder memories.
• Employ a TransCenter Decoder with Deformable Cross-Attention for both Tracking (TDCA) and Detection (DDCA).
• Leverage sparse tracking queries with prior frame positions to compute object displacements through time.
• Output branches compute center heatmaps, object sizes, and tracking displacements; use a center heatmap for detections and a tracking branch for temporally associating objects.
• Train with a center heatmap focal loss, sparse regression loss for sizes, tracking loss, and an overall weighted loss.

Experimental results

Research questions

• RQ1 Can a transformer-based MOT model with dense image-related detection queries outperform sparse-query DETR-based MOT approaches?
• RQ2 Does separating dense detection queries from sparse tracking queries improve detection robustness and tracking efficiency in crowded scenes?
• RQ3 How do different QLN and decoder designs affect MOT accuracy and efficiency?
• RQ4 What is the impact of using efficient encoders (PVT) and deformable attention on MOT runtime and performance?

Key findings

• TransCenter sets new state-of-the-art MOT performance on MOT17 (+4.0% MOTA) and MOT20 (+18.8% MOTA) under their reported conditions.
• Dense image-related detection queries reduce miss detections and noise compared to fixed sparse queries in crowded scenes.
• Sparse tracking queries, driven by prior frame information, significantly speed up tracking attention without sacrificing accuracy.
• Variants TransCenter-Dual and TransCenter-Lite provide trade-offs between accuracy and computational efficiency.