[Paper Review] Cross-Modality Paired-Images Generation for RGB-Infrared Person Re-Identification

Guan-An Wang, Tianzhu Zhang. Yang Yang+4 more|arXiv (Cornell University)|Feb 10, 2020

Video Surveillance and Tracking Methods24 references37 citations

TL;DR

This paper introduces JSIA-ReID, which jointly achieves set-level and instance-level alignment for RGB-IR person re-identification by generating cross-modality paired images through feature disentanglement and exchanging features, improving RGB-IR matching.

ABSTRACT

RGB-Infrared (IR) person re-identification is very challenging due to the large cross-modality variations between RGB and IR images. The key solution is to learn aligned features to the bridge RGB and IR modalities. However, due to the lack of correspondence labels between every pair of RGB and IR images, most methods try to alleviate the variations with set-level alignment by reducing the distance between the entire RGB and IR sets. However, this set-level alignment may lead to misalignment of some instances, which limits the performance for RGB-IR Re-ID. Different from existing methods, in this paper, we propose to generate cross-modality paired-images and perform both global set-level and fine-grained instance-level alignments. Our proposed method enjoys several merits. First, our method can perform set-level alignment by disentangling modality-specific and modality-invariant features. Compared with conventional methods, ours can explicitly remove the modality-specific features and the modality variation can be better reduced. Second, given cross-modality unpaired-images of a person, our method can generate cross-modality paired images from exchanged images. With them, we can directly perform instance-level alignment by minimizing distances of every pair of images. Extensive experimental results on two standard benchmarks demonstrate that the proposed model favourably against state-of-the-art methods. Especially, on SYSU-MM01 dataset, our model can achieve a gain of 9.2% and 7.7% in terms of Rank-1 and mAP. Code is available at https://github.com/wangguanan/JSIA-ReID.

Motivation & Objective

• Motivate RGB-IR Re-ID by addressing large cross-modality gaps between RGB and IR images.
• Propose a method to generate cross-modality paired-images to enable instance-level alignment.
• Disentangle modality-invariant content and modality-specific style features for set-level gap reduction.
• Enable end-to-end learning of modality-aligned and identity-discriminative features.
• Demonstrate state-of-the-art performance on SYSU-MM01 and RegDB benchmarks.

Proposed method

• Propose a generation module G with three encoders (modality-invariant E^i, RGB-specific E^s_rgb, IR-specific E^s_ir) and two decoders to generate cross-modality paired images by exchanging disentangled features.
• Train a feature alignment module F that uses a shared modality-invariant encoder for set-level alignment and an instance-level encoder to minimize distances between paired cross-modality images.
• Apply reconstruction loss, cycle-consistency loss, and GAN loss to ensure realistic cross-modality paired image generation.
• Perform instance-level alignment by minimizing KL-divergences between cross-modality paired-image features in a learned space T using an identity classifier and triplet loss.
• Combine losses into a joint objective L with weights for cycle-consistency, GAN, alignment, and Re-ID losses to learn modality-aligned and identity-discriminative features.
• During testing, extract features via set-level and instance-level encoders and compute cosine similarities for matching.

Experimental results

Research questions

• RQ1 Can disentangling modality-invariant content from modality-specific style reduce set-level cross-modality gaps between RGB and IR images?
• RQ2 Can generated cross-modality paired-images enable effective instance-level alignment for RGB-IR Re-ID?
• RQ3 Does joint set-level and instance-level alignment outperform methods that use only global distribution alignment or only image-to-image translation?
• RQ4 Do the proposed encoders and decoding scheme produce realistic cross-modality images suitable for Re-ID learning?
• RQ5 What is the impact of disentanglement and alignment components on performance on SYSU-MM01 and RegDB?

Key findings

• The proposed JSIA-ReID achieves notable gains over state-of-the-art on SYSU-MM01 (Rank-1 and mAP improvements) and RegDB (Rank-1 and mAP improvements).
• Ablation shows that both set-level and instance-level alignment contribute to performance, with joint SL+IL producing the best results.
• Feature disentanglement aids set-level alignment and reduces modality gap more effectively than non-disentangled baselines.
• Generated cross-modality paired-images enable precise instance-level alignment by directly minimizing distances between image pairs.
• The method demonstrates robust performance across different evaluation settings (single-shot, multi-shot, all-search/ indoor-search).
• Visual analysis indicates the proposed generation yields more faithful cross-modality paired-images compared to CycleGAN/StarGAN baselines.