[Paper Review] Online Multiple Object Tracking with Cross-Task Synergy

Song Guo, Jingya Wang+2 more|arXiv (Cornell University)|Apr 1, 2021

Video Surveillance and Tracking Methods61 references4 citations

TL;DR

This paper proposes a unified online multiple object tracking framework that achieves synergistic improvement between position prediction and identity embedding association through temporal-aware attention and identity-aware memory aggregation. By jointly optimizing both tasks, the model significantly enhances robustness to occlusions, achieving state-of-the-art performance on MOTChallenge benchmarks with a 1.0 MOTA gain over baseline methods.

ABSTRACT

Modern online multiple object tracking (MOT) methods usually focus on two directions to improve tracking performance. One is to predict new positions in an incoming frame based on tracking information from previous frames, and the other is to enhance data association by generating more discriminative identity embeddings. Some works combined both directions within one framework but handled them as two individual tasks, thus gaining little mutual benefits. In this paper, we propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model. Specifically, the attention modules can make the prediction focus more on targets and less on distractors, therefore more reliable embeddings can be extracted accordingly for association. On the other hand, such reliable embeddings can boost identity-awareness through memory aggregation, hence strengthen attention modules and suppress drifts. In this way, the synergy between position prediction and embedding association is achieved, which leads to strong robustness to occlusions. Extensive experiments demonstrate the superiority of our proposed model over a wide range of existing methods on MOTChallenge benchmarks. Our code and models are publicly available at https://github.com/songguocode/TADAM.

Motivation & Objective

• To address the limitation of existing online MOT methods that treat position prediction and embedding association as isolated tasks, leading to poor performance under occlusions.
• To establish mutual benefits between position prediction and data association by linking them through attention mechanisms and memory aggregation.
• To improve tracking robustness in complex scenarios with heavy occlusions, where current methods often fail due to drifting predictions and noisy embeddings.
• To develop a unified, end-to-end trainable model that jointly optimizes both tasks with shared representations.

Proposed method

• Introduces temporal-aware target attention (TA) and distractor attention (DA) modules that use identity embeddings to guide feature focus, reducing interference from nearby objects.
• Employs an identity-aware memory aggregation module that accumulates discriminative embeddings over time to strengthen attention generation and improve prediction reliability.
• Uses the attention modules to refine bounding box predictions by emphasizing targets and suppressing distractors, especially during occlusions.
• Leverages reliable embeddings from attention-guided feature extraction to further improve attention quality, forming a positive feedback loop.
• Jointly optimizes position prediction, embedding association, and all components in an end-to-end manner using a unified loss function.
• Applies adaptive weighting of attention modules based on occlusion levels, applying stronger attention only in high-occlusion scenarios to avoid degradation in easy cases.

Experimental results

Research questions

• RQ1 Can position prediction and embedding association in online MOT be jointly optimized to achieve mutual benefits rather than being treated as independent tasks?
• RQ2 How can attention mechanisms be designed to make position prediction more robust to occlusions by focusing on targets and suppressing distractors?
• RQ3 To what extent does identity-aware memory aggregation improve attention quality and tracking performance under long-term occlusions?
• RQ4 Does adaptive application of attention modules based on occlusion levels lead to better overall performance than uniform application?
• RQ5 Can a unified end-to-end model outperform existing state-of-the-art methods on standard MOT benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on MOTChallenge benchmarks, outperforming existing methods with a significant MOTA gain.
• On occlusion levels above 90%, the tracker achieves a 25% tracked ratio, compared to only 5% for Tracktor++V2, demonstrating strong robustness under extreme occlusions.
• Removing adaptive weighting of attention modules results in a 1.0 MOTA drop, confirming the necessity of context-aware attention application.
• Replacing memory aggregation with simple frame-by-frame reference storage causes a 0.5 MOTA decrease, highlighting the importance of long-term identity representation.
• The ablation study confirms that both target and distractor attention modules contribute significantly to performance, especially in high-occlusion scenarios.
• The synergy between prediction and association is validated through ablation, showing that joint optimization outperforms isolated task optimization.