[Paper Review] Multi-Camera Multi-Object Tracking on the Move via Single-Stage Global Association Approach

Pha Nguyen, Kha Gia Quach+4 more|arXiv (Cornell University)|Nov 17, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a single-stage global association approach for multi-camera multi-object tracking on autonomous vehicles, using Fractional Optimal Transport Assignment (FOTA) to directly link detections across cameras to a single tracked object, reducing ID switches and improving tracking accuracy. The method achieves a 6.4% performance boost and reduces ID switch errors from 3,807 to 870 on the nuScenes test set compared to prior vision-based methods.

ABSTRACT

The development of autonomous vehicles generates a tremendous demand for a low-cost solution with a complete set of camera sensors capturing the environment around the car. It is essential for object detection and tracking to address these new challenges in multi-camera settings. In order to address these challenges, this work introduces novel Single-Stage Global Association Tracking approaches to associate one or more detection from multi-cameras with tracked objects. These approaches aim to solve fragment-tracking issues caused by inconsistent 3D object detection. Moreover, our models also improve the detection accuracy of the standard vision-based 3D object detectors in the nuScenes detection challenge. The experimental results on the nuScenes dataset demonstrate the benefits of the proposed method by outperforming prior vision-based tracking methods in multi-camera settings.

Motivation & Objective

• To address the instability and fragmentation in multi-camera multi-object tracking (MC-MOT) on moving vehicles due to inconsistent 3D object detection across cameras.
• To eliminate the need for two-stage tracking pipelines that rely on separate single-camera tracking and subsequent global matching.
• To improve tracking robustness and accuracy in dynamic MC-MOT settings by directly associating multi-camera detections with global object tracks in a single stage.
• To reduce ID switch errors and improve detection and tracking performance using a unified, end-to-end framework on the nuScenes dataset.

Proposed method

• Proposes a one-to-many global association problem formulation that maps one tracked object in world coordinates to multiple detections across overlapping camera views.
• Introduces Fractional Optimal Transport Assignment (FOTA), a differentiable method to solve the global association as a single-stage optimization problem.
• Develops two SAGA-based frameworks: SAGA-Track (track-by-detection with FOTA) and SAGA-TrackNet (end-to-end track-by-attention with self- and cross-attention layers for cross-camera feature modeling).
• Uses appearance and motion features for cross-camera matching, enhancing robustness to partial or missing detections.
• Employs a unified architecture that jointly learns feature encoding, object localization, and global track association in an end-to-end manner.
• Applies empirical heuristics (e.g., IOU thresholding, box merging) in baseline comparisons for fair evaluation against existing vision-only methods.

Experimental results

Research questions

• RQ1 Can a single-stage global association approach outperform traditional two-stage tracking pipelines in dynamic multi-camera multi-object tracking on autonomous vehicles?
• RQ2 To what extent can FOTA improve tracking accuracy and reduce ID switch errors compared to existing vision-based methods on the nuScenes benchmark?
• RQ3 How effective is the integration of self- and cross-attention mechanisms in modeling cross-camera relationships for global object association?
• RQ4 Does end-to-end learning in SAGA-TrackNet lead to better generalization and robustness than stage-wise tracking with post-hoc global matching?

Key findings

• The proposed SAGA-Track and SAGA-TrackNet methods reduce ID switch errors from 3,807 (baseline) to 870 on the nuScenes test set, representing a significant improvement in tracking stability.
• SAGA-TrackNet achieves an AMOTA of 0.242 and a MOTAR of 0.627, outperforming SOTA vision-only methods like QD-3DT and DEFT in most metrics.
• The method improves tracking performance by up to 6.4% compared to prior vision-based tracking approaches on the nuScenes benchmark.
• SAGA-TrackNet reduces FRAG (fragmentation) errors to 2,000, significantly lower than DEFT’s 3,420, indicating better track continuity.
• The framework demonstrates superior performance in AMOTA, MOTA, and Recall, confirming enhanced accuracy in both object association and bounding box regression.
• The ablation shows that global appearance matching across cameras is a key factor in reducing ID switches and improving tracking robustness.