[Paper Review] Learning Modal-Invariant and Temporal-Memory for Video-based Visible-Infrared Person Re-Identification

Xinyu Lin, Jinxing Li+6 more|arXiv (Cornell University)|Aug 4, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes a novel video-based visible-infrared (RGB-IR) person re-identification method, MITML, which learns modal-invariant features via adversarial training and captures motion-invariant temporal memory through a temporal memory refinement module. On the newly constructed HITSZ-VCM dataset, MITML achieves state-of-the-art performance, with 64.62% R1 and 47.32% mAP in infrared-to-visible retrieval, significantly outperforming existing image-based methods.

ABSTRACT

Thanks for the cross-modal retrieval techniques, visible-infrared (RGB-IR) person re-identification (Re-ID) is achieved by projecting them into a common space, allowing person Re-ID in 24-hour surveillance systems. However, with respect to the probe-to-gallery, almost all existing RGB-IR based cross-modal person Re-ID methods focus on image-to-image matching, while the video-to-video matching which contains much richer spatial- and temporal-information remains under-explored. In this paper, we primarily study the video-based cross-modal person Re-ID method. To achieve this task, a video-based RGB-IR dataset is constructed, in which 927 valid identities with 463,259 frames and 21,863 tracklets captured by 12 RGB/IR cameras are collected. Based on our constructed dataset, we prove that with the increase of frames in a tracklet, the performance does meet more enhancement, demonstrating the significance of video-to-video matching in RGB-IR person Re-ID. Additionally, a novel method is further proposed, which not only projects two modalities to a modal-invariant subspace, but also extracts the temporal-memory for motion-invariant. Thanks to these two strategies, much better results are achieved on our video-based cross-modal person Re-ID. The code and dataset are released at: https://github.com/VCMproject233/MITML.

Motivation & Objective

• To address the lack of video-based cross-modal person Re-ID methods that leverage both spatial and temporal information in visible and infrared modalities.
• To construct a large-scale, video-based RGB-IR person Re-ID dataset (HITSZ-VCM) with 927 identities and 463,259 frames across 12 cameras.
• To develop a method that learns modal-invariant features to reduce domain gap between RGB and IR modalities.
• To extract motion-invariant temporal representations for improved video-level feature learning.
• To demonstrate the superiority of video-based over image-based cross-modal Re-ID through empirical evaluation.

Proposed method

• A novel adversarial learning strategy is used to project RGB and IR features into a shared, modal-invariant subspace, reducing modality-specific bias.
• A temporal memory refinement (TMR) module is designed to capture motion consistency across video frames, enhancing discriminative temporal modeling.
• The TMR module uses a memory bank to store and update temporal features, with a learnable attention mechanism to refine motion-invariant representations.
• A multi-task loss function combines triplet loss and contrastive loss, with a hyperparameter λ to balance the two, optimized via end-to-end training.
• The model is trained and evaluated on a new video-based RGB-IR dataset (HITSZ-VCM), with 11,785 RGB and 10,078 IR tracklets from 12 synchronized cameras.
• Ablation studies validate the effectiveness of modal-invariant learning and temporal memory refinement, with optimal performance at λ=0.4.

Experimental results

Research questions

• RQ1 Does video-based cross-modal Re-ID significantly improve performance over image-based methods in visible-infrared person re-identification?
• RQ2 Can adversarial learning effectively reduce the domain gap between RGB and infrared modalities in video Re-ID?
• RQ3 How effective is a temporal memory refinement module in capturing motion-invariant features for video Re-ID?
• RQ4 What is the optimal balance between triplet and contrastive loss in the proposed MITML framework?
• RQ5 Is the proposed method robust to variations in tracklet length and modality imbalance?

Key findings

• The HITSZ-VCM dataset contains 927 identities, 463,259 frames, and 21,863 tracklets, with 500 identities for training and 427 for testing, significantly larger than existing image-based datasets.
• Video-based Re-ID with longer tracklets yields better performance, confirming the value of temporal information in cross-modal matching.
• MITML achieves 64.62% R1 and 47.32% mAP in infrared-to-visible retrieval, outperforming the second-best method (CAJL) by 7.15% in R1 and 3.82% in mAP.
• The optimal loss weight λ=0.4 yields the best performance, with results stable across a range of λ values, indicating model robustness.
• Max pooling and average pooling outperform weighted pooling in the baseline, suggesting simpler aggregation is more effective for video-level features.
• The ablation study confirms that both modal-invariant learning and temporal memory refinement are essential for performance gains.