[论文解读] Cross-Spectrum Dual-Subspace Pairing for RGB-infrared Cross-Modality Person Re-Identification

Xing Fan, Hao Luo+2 more|arXiv (Cornell University)|Feb 29, 2020

Video Surveillance and Tracking Methods参考文献 53被引用 14

一句话总结

本文提出了一种名为跨光谱双子空间配对（Cross-Spectrum Dual-Subspace Pairing, CDP）的一流深度学习模型，用于RGB-红外行人重识别，通过多光谱图像生成与双子空间配对策略增强跨模态特征学习。通过生成合成多光谱图像，并将真实样本与生成样本配对以减少模态差异，CDP实现了最先进性能，在RegDB数据集上将rank-1准确率提升至65.0%，mAP提升至62.7%。

ABSTRACT

Due to its potential wide applications in video surveillance and other computer vision tasks like tracking, person re-identification (ReID) has become popular and been widely investigated. However, conventional person re-identification can only handle RGB color images, which will fail at dark conditions. Thus RGB-infrared ReID (also known as Infrared-Visible ReID or Visible-Thermal ReID) is proposed. Apart from appearance discrepancy in traditional ReID caused by illumination, pose variations and viewpoint changes, modality discrepancy produced by cameras of the different spectrum also exists, which makes RGB-infrared ReID more difficult. To address this problem, we focus on extracting the shared cross-spectrum features of different modalities. In this paper, a novel multi-spectrum image generation method is proposed and the generated samples are utilized to help the network to find discriminative information for re-identifying the same person across modalities. Another challenge of RGB-infrared ReID is that the intra-person (images from the same person) discrepancy is often larger than the inter-person (images from different persons) discrepancy, so a dual-subspace pairing strategy is proposed to alleviate this problem. Combining those two parts together, we also design a one-stream neural network combining the aforementioned methods to extract compact representations of person images, called Cross-spectrum Dual-subspace Pairing (CDP) model. Furthermore, during the training process, we also propose a Dynamic Hard Spectrum Mining method to automatically mine more hard samples from hard spectrum based on the current model state to further boost the performance. Extensive experimental results on two public datasets, SYSU-MM01 with RGB + near-infrared images and RegDB with RGB + far-infrared images, have demonstrated the efficiency and generality of our proposed method.

研究动机与目标

• 为解决低光照条件下RGB与红外图像之间存在的模态差异问题。
• 通过学习共享的跨光谱特征，减少因光谱差异导致的跨人距离过大（即内人距离常超过跨人距离）的问题。
• 通过统一的一流网络架构，提升特征的紧凑性与判别能力。
• 通过基于模型状态动态挖掘难样本，提升训练效率与泛化能力。
• 验证跨光谱图像生成在丰富训练多样性与提升模态间特征对齐方面的有效性。

提出的方法

• 一种跨光谱图像生成方法通过结合RGB与红外图像，合成多光谱训练样本，从而在训练过程中提升模态多样性。
• 一种双子空间配对策略将同一人的真实图像与另一光谱下的生成图像进行配对，使网络更关注模态不变特征。
• 一种一流卷积神经网络整合了图像生成与配对策略，以学习跨模态的紧凑共享表征。
• 动态难光谱样本挖掘机制根据当前模型性能自动识别并优先处理难样本，从而提升特征判别能力。
• 模型采用PK采样策略结合三元组损失函数优化特征嵌入，t-SNE可视化结果证实了特征聚类的显著改善。
• 该方法在公开数据集上端到端训练，消融实验验证了RGB通道同质性的必要性，且YUV与HSV等替代色彩空间在该跨光谱框架下均无法收敛。

实验结果

研究问题

• RQ1 多光谱图像生成是否能通过提升训练模态多样性，改善RGB-红外行人重识别中的特征学习？
• RQ2 双子空间配对策略是否能通过聚焦模态不变特征，有效减少内人距离？
• RQ3 在跨模态ReID任务中，一流网络架构是否能以更简单的训练流程超越两流或两阶段方法？
• RQ4 动态难光谱样本挖掘是否能通过根据模型状态自适应调整，提升模型泛化能力？
• RQ5 为何在该跨光谱框架中，YUV与HSV等替代色彩空间无法收敛？

主要发现

• 在RegDB数据集上，所提出的CDP模型实现了65.0%的rank-1准确率与62.7%的mAP，显著优于此前最先进方法的50.9%与47.0%。
• 在SYSU-MM01数据集上，CDP实现了36.2%的rank-1准确率与37.5%的mAP，表明其在多个数据集上均保持一致的性能提升。
• t-SNE可视化结果表明，CDP能将同一人的多光谱图像映射到统一的特征分布中，有效降低内人距离并增强跨人分离度。
• 消融实验显示，若使用YUV或HSV色彩空间替代RGB，则因通道异质性与预训练不兼容导致训练失败。
• 跨光谱图像生成方法在模态数量增加时性能持续提升，表明更丰富的光谱多样性有助于提升特征学习效果。
• 动态难光谱样本挖掘策略通过自适应模型当前状态并聚焦于困难样本，有效提升了模型性能。