[Paper Review] CityFlow-NL: Tracking and Retrieval of Vehicles at City Scale by Natural Language Descriptions

Qi Feng, Vitaly Ablavsky+1 more|arXiv (Cornell University)|Jan 12, 2021

Human Mobility and Location-Based AnalysisSocial Sciences39 references27 citations

TL;DR

CityFlow-NL introduces a city-scale multi-target multi-camera tracking benchmark with natural language descriptions, and offers retrieval and tracking baselines plus a novel one-shot NL-aware tracker (VTN).

ABSTRACT

Natural Language (NL) descriptions can be one of the most convenient or the only way to interact with systems built to understand and detect city scale traffic patterns and vehicle-related events. In this paper, we extend the widely adopted CityFlow Benchmark with NL descriptions for vehicle targets and introduce the CityFlow-NL Benchmark. The CityFlow-NL contains more than 5,000 unique and precise NL descriptions of vehicle targets, making it the first multi-target multi-camera tracking with NL descriptions dataset to our knowledge. Moreover, the dataset facilitates research at the intersection of multi-object tracking, retrieval by NL descriptions, and temporal localization of events. In this paper, we focus on two foundational tasks: the Vehicle Retrieval by NL task and the Vehicle Tracking by NL task, which take advantage of the proposed CityFlow-NL benchmark and provide a strong basis for future research on the multi-target multi-camera tracking by NL description task.

Motivation & Objective

• Extend CityFlow with natural language descriptions for vehicle targets to enable MTMC tracking, retrieval by NL, and temporal event localization.
• Define and evaluate two foundational tasks: Vehicle Retrieval by NL Description and Vehicle Tracking by NL Description.
• Provide baselines and a dedicated NL-guided tracking model to pave the way for MTMC by NL research.

Proposed method

• Create CityFlow-NL by annotating 666 target vehicles with 5,289 NL descriptions across 40 calibrated cameras.
• Propose two tasks: Vehicle Retrieval by NL (single-view) and Vehicle Tracking by NL (multi-frame localization).
• Develop a baseline Siamese retrieval model that embeds NL queries (BERT) and vehicle crops (ResNet-50) to compute similarity for retrieval.
• Extend to a tracking framework: baseline track-then-retrieve using multi-object trackers; introduce the Vehicle Tracking Network (VTN) with a presence branch and a localization branch.
• Introduce NL-RoI Head and NL similarity learning in a Faster-RCNN-based localization pipeline for end-to-end NL-guided tracking.
• Evaluate using MRR, Recall@K for retrieval, and AUC of Success vs IoU thresholds plus Normalized Precision for tracking.

Experimental results

Research questions

• RQ1 How well can NL descriptions retrieve the correct vehicle tracks in single-view video?
• RQ2 Can NL descriptions support temporal and spatial localization of targets in multi-view videos?
• RQ3 Does an NL-guided end-to-end tracker (VTN) outperform track-then-retrieve baselines in city-scale MTMC scenarios?

Key findings

• CityFlow-NL is the first MTMC-by-NL dataset and largest NL-annotated tracking benchmark by number of NL descriptions (5,289 NL descriptions across 666 targets).
• Baseline retrieval yields limited performance (MRR ≈ 0.027, Recall@5 ≈ 0.026, Recall@10 ≈ 0.049, Recall@25 ≈ 0.111).
• VTN (Vehicle Tracking Network) achieves notable improvement with an AUC of 5.93 and normalized precision of 3.79 on the test split, outperforming track-then-retrieve baselines.
• VLN NL-based RoI localization and NL-based presence scoring enable online, end-to-end NL-guided tracking at ~20 fps on a single GPU.
• VTN shows higher recall due to early NL retrieval during detection but incurs higher false positives in scenes with contrasting targets.