[Paper Review] Enhancing the Association in Multi-Object Tracking via Neighbor Graph

Tianyi Liang, Long Lan+1 more|arXiv (Cornell University)|Jul 1, 2020

Video Surveillance and Tracking Methods39 references4 citations

TL;DR

This paper proposes a neighbor graph-based method to enhance data association in multi-object tracking by leveraging spatio-temporal relationships among nearby targets to improve robustness against occlusions, illumination changes, and detection errors. By constructing a graph of a target and its selected neighbors and applying graph convolutional networks (GCN), the approach learns more reliable, group-aware features, achieving state-of-the-art performance on MOT benchmarks under online tracking protocols.

ABSTRACT

Most modern multi-object tracking (MOT) systems follow the tracking-by-detection paradigm. It first localizes the objects of interest, then extracting their individual appearance features to make data association. The individual features, however, are susceptible to the negative effects as occlusions, illumination variations and inaccurate detections, thus resulting in the mismatch in the association inference. In this work, we propose to handle this problem via making full use of the neighboring information. Our motivations derive from the observations that people tend to move in a group. As such, when an individual target's appearance is seriously changed, we can still identify it with the help of its neighbors. To this end, we first utilize the spatio-temporal relations produced by the tracking self to efficiently select suitable neighbors for the targets. Subsequently, we construct neighbor graph of the target and neighbors then employ the graph convolution networks (GCN) to learn the graph features. To the best of our knowledge, it is the first time to exploit neighbor cues via GCN in MOT. Finally, we test our approach on the MOT benchmarks and achieve state-of-the-art performance in online tracking.

Motivation & Objective

• To address the vulnerability of individual appearance features in multi-object tracking under occlusions, illumination variations, and inaccurate detections.
• To improve data association robustness by exploiting contextual information from neighboring targets.
• To develop an efficient, lightweight neighbor selection strategy using spatio-temporal cues from the tracking system itself.
• To integrate graph convolution networks (GCN) to learn collective features from target-neighbor relationships for improved tracking performance.

Proposed method

• Spatio-temporal relations from the tracking system are used to select relevant neighbors for each target, ensuring efficiency and relevance.
• A neighbor graph is constructed with the target as the central node and its selected neighbors as connected nodes.
• Graph Convolutional Networks (GCN) are applied to aggregate and refine features across the neighbor graph, learning more robust, context-aware representations.
• The neighbor graph module is integrated into the JDE framework, with GCN layers added after detection and embedding heads.
• The number of neighbors K is set to 4, and when fewer than K neighbors are available, the target is copied to fill the graph.
• The model is trained end-to-end using Adam optimizer with learning rate decay, data augmentation, and pretraining on COCO and CUHK-SYSU datasets.

Experimental results

Research questions

• RQ1 Can neighboring targets provide reliable contextual cues to improve data association when a target's appearance is degraded by occlusion or noise?
• RQ2 How can we efficiently and effectively select relevant neighbors for a target using only spatio-temporal information from the tracking system?
• RQ3 To what extent can graph convolutional networks enhance feature representation by modeling relationships between a target and its neighbors in multi-object tracking?
• RQ4 Does incorporating neighbor graph features lead to measurable improvements in MOTA and IDF1 under online tracking protocols?

Key findings

• The proposed method achieves a MOTA of 57.7 and IDF1 of 62.6 on the MOT16 benchmark under the public detection protocol, outperforming all prior online trackers.
• On MOT17, the method achieves a MOTA of 58.4 and IDF1 of 62.9, demonstrating state-of-the-art performance among online trackers.
• The neighbor graph module improves the JDE baseline by increasing MOTA from 76.4 to 76.6 and reducing IDs from 2271 to 2049 on MOT17 training set.
• The method significantly reduces id-switches (IDs) and false positives, indicating more stable and accurate data association.
• The ablation study confirms that neighbor graph learning effectively mitigates errors caused by appearance degradation and detection noise.
• The approach is effective even when neighbor count is low, as the model handles sparse graphs by copying the target to fill missing neighbors.