[Paper Review] ReST: A Reconfigurable Spatial-Temporal Graph Model for Multi-Camera Multi-Object Tracking

Cheng-Che Cheng, Min-Xuan Qiu+2 more|arXiv (Cornell University)|Aug 25, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes ReST, a reconfigurable spatial-temporal graph model for multi-camera multi-object tracking that decouples association into spatial and temporal stages to improve robustness to occlusion and ID switching. By reconfiguring graphs iteratively to merge consistent detections across views and frames, ReST achieves state-of-the-art performance on Wildtrack and strong results on other benchmarks without relying on single-camera tracker outputs.

ABSTRACT

Multi-Camera Multi-Object Tracking (MC-MOT) utilizes information from multiple views to better handle problems with occlusion and crowded scenes. Recently, the use of graph-based approaches to solve tracking problems has become very popular. However, many current graph-based methods do not effectively utilize information regarding spatial and temporal consistency. Instead, they rely on single-camera trackers as input, which are prone to fragmentation and ID switch errors. In this paper, we propose a novel reconfigurable graph model that first associates all detected objects across cameras spatially before reconfiguring it into a temporal graph for Temporal Association. This two-stage association approach enables us to extract robust spatial and temporal-aware features and address the problem with fragmented tracklets. Furthermore, our model is designed for online tracking, making it suitable for real-world applications. Experimental results show that the proposed graph model is able to extract more discriminating features for object tracking, and our model achieves state-of-the-art performance on several public datasets.

Motivation & Objective

• Address the limitations of single-camera trackers in handling occlusion and ID switching in crowded scenes.
• Overcome the reliance on pre-generated tracklets from single-camera trackers in existing graph-based MC-MOT methods.
• Improve feature representation by explicitly modeling spatial and temporal consistency in a two-stage graph framework.
• Enable online tracking by avoiding future frame dependency through iterative graph reconfiguration.
• Enhance generalization across datasets by reducing dependence on appearance features under varying lighting and motion conditions.

Proposed method

• Formulate MC-MOT as two sub-tasks: Spatial Association (cross-camera matching at the same time step) and Temporal Association (cross-frame matching).
• Construct a spatial graph using detections across cameras at the same frame, with edge features based on appearance and geometric consistency.
• Apply a Graph Reconfiguration module that merges nodes of the same object into connected components, simplifying the graph for temporal association.
• Reconfigure the graph into a temporal graph by connecting nodes across frames, enabling temporal feature learning and refinement.
• Train spatial and temporal graphs separately with task-specific constraints to improve feature discrimination and optimization stability.
• Iteratively reconfigure the graph during inference to refine associations and enhance feature representation in dynamic scenes.

Experimental results

Research questions

• RQ1 Can a two-stage graph-based approach that separates spatial and temporal association improve tracking robustness in multi-camera settings?
• RQ2 How does graph reconfiguration enhance feature representation and reduce fragmentation in multi-object tracking?
• RQ3 To what extent can the model generalize across datasets without heavy reliance on appearance features?
• RQ4 Does decoupling spatial and temporal learning lead to better optimization and performance than unified spatial-temporal graph models?
• RQ5 Can the proposed method achieve state-of-the-art performance in online multi-camera multi-object tracking without future frame dependency?

Key findings

• ReST achieves state-of-the-art performance on the Wildtrack dataset with an IDF1 of 91.6 and MOTA of 97.0, outperforming existing methods.
• On Wildtrack, the model achieves 91.6 IDF1 and 97.0 MOTA when using appearance, projection, and speed features, demonstrating robustness to appearance variation.
• Removing appearance features reduces performance to 89.2 IDF1 and 95.0 MOTA, indicating that ReST generalizes well even without strong appearance cues.
• The unified spatial-temporal graph model performs worse than the separated spatial-temporal graphs, as shown by t-SNE visualization revealing poor separation between positive and negative edges.
• ReST shows strong cross-dataset generalization: fine-tuning on Wildtrack and testing on CAMPUS improves MOTA by up to 15.5 percentage points.
• t-SNE visualization confirms that ReST learns more discriminative node features with better inter-class separation and intra-class clustering than standard ReID features.