[Paper Review] DeepMOT: A Differentiable Framework for Training Multiple Object Trackers

Yihong Xu, Yutong Ban+2 more|arXiv (Cornell University)|Jun 15, 2019

Video Surveillance and Tracking Methods27 references29 citations

TL;DR

This paper proposes DeepMOT, a differentiable framework that enables end-to-end training of deep multi-object trackers by introducing differentiable proxies for MOTA and MOTP using a Deep Hungarian Net (DHN) module. The method improves tracker performance and establishes a new state of the art on the MOTChallenge benchmark.

ABSTRACT

The recent trend in vision-based multi-object tracking (MOT) is heading towards leveraging the representational power of deep learning to jointly learn to detect and track objects. However, existing methods train only certain sub-modules using loss functions that often do not correlate with established tracking evaluation measures such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP). As these measures are not differentiable, the choice of appropriate loss functions for end-to-end training of multi-object tracking methods is still an open research problem. In this paper, we bridge this gap by proposing a differentiable proxy of MOTA and MOTP, which we combine in a loss function suitable for end-to-end training of deep multi-object trackers. As a key ingredient, we propose a Deep Hungarian Net (DHN) module that approximates the Hungarian matching algorithm. DHN allows estimating the correspondence between object tracks and ground truth objects to compute differentiable proxies of MOTA and MOTP, which are in turn used to optimize deep trackers directly. We experimentally demonstrate that the proposed differentiable framework improves the performance of existing multi-object trackers, and we establish a new state of the art on the MOTChallenge benchmark. Our code is publicly available from this https URL.

Motivation & Objective

• To address the challenge of training multi-object trackers using loss functions that do not correlate with standard tracking evaluation metrics like MOTA and MOTP.
• To bridge the gap between non-differentiable tracking metrics and deep learning optimization by creating differentiable proxies for MOTA and MOTP.
• To enable end-to-end training of deep multi-object trackers using a loss function grounded in actual tracking performance.
• To develop a differentiable approximation of the Hungarian matching algorithm for correspondence estimation between tracks and ground truth objects.

Proposed method

• Proposes a differentiable proxy for MOTA and MOTP by approximating the Hungarian matching algorithm using a learnable module called the Deep Hungarian Net (DHN).
• Uses the DHN module to estimate track-to-ground-truth object correspondences in a differentiable manner, enabling gradient flow through the matching process.
• Combines the differentiable MOTA and MOTP proxies into a single loss function for end-to-end optimization of deep multi-object trackers.
• Integrates the differentiable loss into existing tracker architectures to allow joint optimization of detection and tracking components.
• Employs a differentiable IoU computation to support the differentiable tracking loss during training.
• Trains the full tracker end-to-end using backpropagation, with gradients flowing through the DHN and tracking loss components.

Experimental results

Research questions

• RQ1 Can MOTA and MOTP, the standard metrics for multi-object tracking, be made differentiable to enable end-to-end training of deep trackers?
• RQ2 How can the Hungarian matching algorithm be approximated in a differentiable way to support gradient-based optimization in multi-object tracking?
• RQ3 Does using differentiable proxies of MOTA and MOTP as a training loss lead to improved tracker performance on standard benchmarks?
• RQ4 Can the proposed framework be integrated into existing deep multi-object tracking architectures without architectural overhaul?

Key findings

• The proposed differentiable framework, DeepMOT, enables end-to-end training of multi-object trackers using a loss function aligned with standard evaluation metrics.
• The Deep Hungarian Net (DHN) module successfully approximates the Hungarian matching algorithm in a differentiable manner, allowing gradient flow through track-to-truth correspondence estimation.
• The method improves the performance of existing multi-object trackers on the MOTChallenge benchmark, achieving a new state of the art.
• The differentiable MOTA and MOTP proxies lead to more effective optimization compared to standard detection and tracking loss combinations.