[Paper Review] AXM-Net: Cross-Modal Context Sharing Attention Network for Person Re-ID.

Ammarah Farooq, Muhammad Awais+2 more|arXiv (Cornell University)|Jan 19, 2021

Video Surveillance and Tracking Methods33 references4 citations

TL;DR

AXM-Net is a CNN-based cross-modal attention network that learns semantically aligned visual and textual representations for person re-identification by leveraging a learnable context-sharing attention mechanism and cross-modal affinity loss. It achieves state-of-the-art performance through end-to-end training and fine-grained attention to local features, significantly outperforming prior methods on both person search and cross-modal Re-ID benchmarks.

ABSTRACT

Cross-modal person re-identification (Re-ID) is critical for modern video surveillance systems. The key challenge is to align inter-modality representations according to semantic information present for a person and ignore background information. In this work, we present AXM-Net, a novel CNN based architecture designed for learning semantically aligned visual and textual representations. The underlying building block consists of multiple streams of feature maps coming from visual and textual modalities and a novel learnable context sharing semantic alignment network. We also propose complementary intra modal attention learning mechanisms to focus on more fine-grained local details in the features along with a cross-modal affinity loss for robust feature matching. Our design is unique in its ability to implicitly learn feature alignments from data. The entire AXM-Net can be trained in an end-to-end manner. We report results on both person search and cross-modal Re-ID tasks. Extensive experimentation validates the proposed framework and demonstrates its superiority by outperforming the current state-of-the-art methods by a significant margin.

Motivation & Objective

• Address the challenge of aligning visual and textual representations in cross-modal person re-identification while suppressing background noise.
• Develop a learnable, end-to-end trainable architecture that implicitly learns feature alignment from data.
• Enhance feature representation by focusing on fine-grained local details through intra-modal attention mechanisms.
• Improve robustness in cross-modal matching using a novel cross-modal affinity loss function.
• Achieve state-of-the-art performance on both person search and cross-modal Re-ID tasks.

Proposed method

• Employ multiple stream feature maps from visual and textual modalities as input to the network.
• Introduce a learnable context-sharing semantic alignment network that dynamically aligns cross-modal features based on shared semantic context.
• Apply complementary intra-modal attention mechanisms to highlight discriminative local features within each modality.
• Design a cross-modal affinity loss to encourage matching between positive pairs and separation of negative pairs in the joint embedding space.
• Train the entire AXM-Net architecture end-to-end to optimize feature alignment and representation learning jointly.
• Leverage the interaction between modality-specific features and shared context to improve semantic consistency across modalities.

Experimental results

Research questions

• RQ1 Can a learnable context-sharing attention mechanism effectively align visual and textual features without explicit supervision?
• RQ2 How does intra-modal attention improve the discriminativeness of local features in cross-modal person re-identification?
• RQ3 To what extent does the proposed cross-modal affinity loss enhance matching robustness compared to standard contrastive losses?
• RQ4 Can end-to-end training of a unified network architecture outperform pipelined or two-stage approaches in cross-modal Re-ID?
• RQ5 Does the model generalize effectively across both person search and cross-modal Re-ID benchmarks?

Key findings

• AXM-Net achieves state-of-the-art performance on both person search and cross-modal Re-ID benchmarks, outperforming existing methods by a significant margin.
• The integration of cross-modal affinity loss leads to more discriminative feature embeddings, improving matching accuracy on hard negative samples.
• Intra-modal attention mechanisms enhance the model’s ability to focus on relevant local features, contributing to improved feature representation.
• The context-sharing attention mechanism enables implicit, data-driven feature alignment, reducing reliance on handcrafted alignment strategies.
• End-to-end training of AXM-Net yields superior generalization and robustness compared to stage-wise training approaches.
• The model demonstrates strong performance across diverse surveillance scenarios, indicating effective alignment of semantic content across modalities.