[Paper Review] SCMM: Calibrating Cross-modal Representations for Text-Based Person Search

Jing Liu, Donglai Wei+6 more|arXiv (Cornell University)|Apr 5, 2023

Video Surveillance and Tracking Methods6 citations

TL;DR

This paper proposes SCMM, a simple yet effective dual-encoder framework for text-based person search that calibrates cross-modal features via two novel losses: Sew calibration loss for fine-grained modality alignment and Masked Caption Modeling (MCM) loss for detailed image-text correspondence. The method achieves state-of-the-art performance with 73.81% Rank-1 accuracy on CUHK-PEDES, 74.25% on ICFG-PEDES, and 57.35% on RSTPReID, enabling high-speed inference without complex modules.

ABSTRACT

Text-Based Person Search (TBPS) aims to retrieve target person images from a large-scale gallery using natural language descriptions, posing fundamental challenges in cross-modal representation learning. Existing methods often struggle to bridge the semantic gap between heterogeneous modalities while capturing fine-grained correspondences essential for discriminating visually similar individuals. To address these challenges, we propose Sew Calibration and Masked Modeling (SCMM), a unified framework that calibrates cross-modal representations through complementary learning mechanisms. Notably, SCMM introduces two novel components: a sew calibration loss that dynamically aligns image-text features using quality-guided adaptive margins based on textual information density, and a masked caption modeling loss that establishes fine-grained cross-modal correspondences through transformer-based masked prediction. Additionally, the sew calibration mechanism implements bidirectional constraints to effectively compress same-identity features in the shared embedding space, while the masked modeling component leverages a cross-modal decoder to learn word-level visual-textual relationships, enabling discrimination of subtle attribute differences. Our dual-encoder architecture achieves an effective balance between representation capability and computational efficiency by employing a training-only decoder design. Extensive experiments on CUHK-PEDES, ICFG-PEDES, and RSTPReID benchmarks demonstrate that SCMM achieves state-of-the-art performance with Rank1 accuracies of 73.81%, 64.25%, and 57.35%, respectively. Comprehensive ablation studies validate the effectiveness of each proposed component.

Motivation & Objective

• To address the challenge of inter-modal gap in cross-modal representation learning for text-based person search (TBPS).
• To improve fine-grained image-text correspondence for better discrimination of similar pedestrian targets.
• To develop a cost-effective, high-speed inference method without multi-level branches or complex interaction modules.
• To enhance feature calibration using caption quality and masked caption prediction for richer cross-modal understanding.

Proposed method

• The framework uses a dual-encoder architecture to extract image and text features, enabling fast inference without extra modules.
• The Sew calibration loss aligns image and text features by leveraging caption quality, pulling positive pairs together and pushing negative pairs apart.
• A cross-modal decoder is introduced to compute the Masked Caption Modeling (MCM) loss, which predicts masked tokens in captions using visual features.
• The MCM loss establishes detailed, fine-grained correspondence between visual parts and textual components through a masked prediction task.
• The decoder is discarded at inference, ensuring no additional computational cost.
• The final retrieval uses only the CLS tokens from the dual-encoder for similarity computation.

Experimental results

Research questions

• RQ1 How can cross-modal features be effectively calibrated to reduce the inter-modal gap in text-based person search?
• RQ2 Can a simple dual-encoder architecture achieve state-of-the-art performance without complex interaction modules or multi-level branches?
• RQ3 To what extent can masked caption prediction improve fine-grained image-text correspondence in TBPS?
• RQ4 How does the quality of textual captions influence the alignment of cross-modal representations?
• RQ5 What is the generalization capability of the proposed method across different TBPS benchmarks?

Key findings

• SCMM achieves 73.81% Rank-1 accuracy on CUHK-PEDES, surpassing previous state-of-the-art methods.
• On ICFG-PEDES, the method reaches 74.25% Rank-1 accuracy, demonstrating strong performance on a diverse dataset.
• On RSTPReID, the model achieves 57.35% Rank-1 accuracy, indicating robustness across domains.
• The MCM loss improves generalization, with a 1.59% Rank-1 gain on ICFG-PEDES and 1.85% on RSTPReID in cross-domain evaluation.
• Visualization shows that the proposed method learns more refined attention maps at both caption-level and word-level, capturing key details like clothing and accessories.
• Feature distribution visualization confirms that image-text features for the same identity are more compact and better separated, reducing the cross-modal gap.