[Paper Review] SBNet: Segmentation-based Network for Natural Language-based Vehicle Search

Sangrok Lee, Taekang Woo+1 more|arXiv (Cornell University)|Apr 22, 2021

Multimodal Machine Learning ApplicationsComputer Science29 references7 citations

TL;DR

SBNet is a segmentation-based deep neural network for natural language-based vehicle retrieval that leverages attention mechanisms and two novel modules—substitution and future prediction—to improve cross-modal alignment and temporal modeling. It achieves a 10th-place ranking in the AI City Challenge 2021 with an MRR of 0.1195, significantly outperforming the baseline model (MRR 0.0269).

ABSTRACT

Natural language-based vehicle retrieval is a task to find a target vehicle within a given image based on a natural language description as a query. This technology can be applied to various areas including police searching for a suspect vehicle. However, it is challenging due to the ambiguity of language descriptions and the difficulty of processing multi-modal data. To tackle this problem, we propose a deep neural network called SBNet that performs natural language-based segmentation for vehicle retrieval. We also propose two task-specific modules to improve performance: a substitution module that helps features from different domains to be embedded in the same space and a future prediction module that learns temporal information. SBnet has been trained using the CityFlow-NL dataset that contains 2,498 tracks of vehicles with three unique natural language descriptions each and tested 530 unique vehicle tracks and their corresponding query sets. SBNet achieved a significant improvement over the baseline in the natural language-based vehicle tracking track in the AI City Challenge 2021.

Motivation & Objective

• To address the challenge of natural language-based vehicle retrieval in surveillance systems where image queries are unavailable.
• To improve cross-modal alignment between natural language descriptions and visual features in multi-modal data.
• To enhance performance through task-specific modules that model domain shift and temporal dynamics in vehicle tracking.
• To develop a segmentation-based network that localizes target vehicles based on NL queries with high spatial precision.

Proposed method

• Proposes SBNet, a segmentation-based network that jointly processes images and natural language descriptions using an image perception module (IPM) and a language modeling module (NLM).
• Integrates a multi-modal fusion module to align visual and textual embeddings in a shared feature space.
• Introduces a substitution module to align features from different domains (e.g., text and image) by learning domain-invariant representations.
• Employs a future prediction module to model temporal motion patterns in vehicle tracks using future frame prediction.
• Uses mean squared error loss for future prediction and cross-entropy loss for classification, with label smoothing to prevent overfitting.
• Applies data denoising via color and type voting to unify inconsistent descriptions in the CityFlow-NL dataset.

Experimental results

Research questions

• RQ1 How can a deep learning model effectively align natural language descriptions with visual features for vehicle retrieval?
• RQ2 What role does temporal modeling play in improving retrieval performance for vehicle tracking with NL queries?
• RQ3 Can domain alignment via a substitution module improve cross-modal embedding quality in multi-modal retrieval?
• RQ4 How do auxiliary modules like future prediction and substitution contribute to performance gains in segmentation-based retrieval?
• RQ5 To what extent can a segmentation-based approach outperform baseline models in NL-based vehicle retrieval?

Key findings

• SBNet achieved an MRR of 0.1195 on the CityFlow-NL benchmark, ranking 10th in the AI City Challenge 2021.
• The substitution module alone improved MRR by 1% over the baseline, demonstrating its effectiveness in cross-domain feature alignment.
• The future prediction module contributed a 0.7% MRR improvement, indicating the value of temporal modeling in vehicle tracking.
• The classification module added a 0.5% MRR gain, showing incremental benefit from auxiliary supervision.
• When all modules were combined, SBNet achieved a 3.5× improvement over the baseline model (MRR 0.0269), confirming the synergistic effect of the proposed components.
• Ablation studies confirmed that each module contributes independently to performance, with the full model outperforming all variants.