[Paper Review] Graph Neural Networks for 3D Multi-Object Tracking

Xinshuo Weng, Yongxin Wang+2 more|arXiv (Cornell University)|Aug 20, 2020

Video Surveillance and Tracking Methods11 references4 citations

TL;DR

This paper proposes a novel 3D multi-object tracking (MOT) method that leverages Graph Neural Networks (GNNs) for feature interaction and a joint 2D/3D feature extractor to improve discriminative feature learning. By modeling object dependencies through GNNs and fusing appearance and motion features from both 2D and 3D spaces, the method achieves state-of-the-art performance on the KITTI dataset, with 93.92 sAMOTA and 86.03 MOTA.

ABSTRACT

3D Multi-object tracking (MOT) is crucial to autonomous systems. Recent work often uses a tracking-by-detection pipeline, where the feature of each object is extracted independently to compute an affinity matrix. Then, the affinity matrix is passed to the Hungarian algorithm for data association. A key process of this pipeline is to learn discriminative features for different objects in order to reduce confusion during data association. To that end, we propose two innovative techniques: (1) instead of obtaining the features for each object independently, we propose a novel feature interaction mechanism by introducing Graph Neural Networks; (2) instead of obtaining the features from either 2D or 3D space as in prior work, we propose a novel joint feature extractor to learn appearance and motion features from 2D and 3D space. Through experiments on the KITTI dataset, our proposed method achieves state-of-the-art 3D MOT performance. Our project website is at http://www.xinshuoweng.com/projects/GNN3DMOT.

Motivation & Objective

• To improve discriminative feature learning in 3D multi-object tracking by modeling dependencies between objects.
• To address the limitation of independent feature extraction in prior tracking-by-detection pipelines.
• To leverage complementary information from both 2D and 3D spaces for more robust motion and appearance features.
• To develop a joint feature extractor that fuses 2D and 3D appearance and motion features for improved data association.
• To apply Graph Neural Networks to enable context-aware feature updates through inter-object interactions.

Proposed method

• Constructs a graph where each node represents an object's fused feature from 2D and 3D space, with edges representing potential associations.
• Applies multi-layer GNNs to update each node’s feature by aggregating information from neighboring nodes, enabling context-aware feature learning.
• Employs a four-branch feature extractor to learn 2D appearance, 2D motion, 3D appearance, and 3D motion features independently before fusion.
• Fuses features from all four branches into a single representation before feeding into the GNN for interaction and affinity computation.
• Uses batch triplet loss on node features and affinity loss on predicted similarity matrices for end-to-end training.
• Performs data association via edge regression in the GNN to predict the final affinity matrix for matching.

Experimental results

Research questions

• RQ1 Can graph neural networks improve discriminative feature learning in 3D multi-object tracking by modeling inter-object dependencies?
• RQ2 Does joint learning of 2D and 3D appearance and motion features lead to more robust and discriminative features than using either modality alone?
• RQ3 How does the proposed GNN-based feature interaction mechanism compare to independent feature extraction in terms of data association accuracy?
• RQ4 Can the fusion of 2D and 3D features reduce confusion between similar objects and improve tracking robustness in complex scenes?
• RQ5 What is the impact of context-aware feature learning on standard MOT metrics like MOTA and AMOTA in 3D tracking?

Key findings

• The proposed method achieves a state-of-the-art sAMOTA score of 93.92 on the KITTI 3D MOT benchmark, surpassing previous methods.
• The method achieves 86.03 MOTA, significantly outperforming mmMOT (74.07), FANTrack (74.30), and AB3DMOT (86.24) in the same evaluation setting.
• The method reduces identity switches (IDS) to 0, indicating perfect identity consistency, compared to 10 for mmMOT and 35 for FANTrack.
• The method achieves 10 fragmentation events (FRAG), the lowest among all compared methods, indicating superior tracking continuity.
• The AMOTA score of 45.83 and AMOTP of 78.10 demonstrate strong performance in both overall tracking accuracy and localization precision.
• Ablation studies confirm that both the joint 2D/3D feature extractor and the GNN-based interaction mechanism are essential for the performance gain.