[论文解读] Learning Granularity-Unified Representations for Text-to-Image Person Re-identification

Zhiyin Shao, Xinyu Zhang+4 more|arXiv (Cornell University)|Jul 16, 2022

Video Surveillance and Tracking Methods被引用 9

一句话总结

该论文提出LGUR，一种基于Transformer的框架，通过将细粒度视觉特征与粗粒度文本特征对齐，学习文本到图像行人重识别的粒度统一表征。该方法引入了多模态共享词典（MSD），在跨模态引导下实现跨模态特征重建，并采用以前景为中心的优化策略，随后通过共享原型实现统一特征提取，在CUHK-PEDES和ICFG-PEDES数据集上分别取得6.58%和6.93%的Rank-1准确率提升，达到当前最先进性能。

ABSTRACT

Text-to-image person re-identification (ReID) aims to search for pedestrian images of an interested identity via textual descriptions. It is challenging due to both rich intra-modal variations and significant inter-modal gaps. Existing works usually ignore the difference in feature granularity between the two modalities, i.e., the visual features are usually fine-grained while textual features are coarse, which is mainly responsible for the large inter-modal gaps. In this paper, we propose an end-to-end framework based on transformers to learn granularity-unified representations for both modalities, denoted as LGUR. LGUR framework contains two modules: a Dictionary-based Granularity Alignment (DGA) module and a Prototype-based Granularity Unification (PGU) module. In DGA, in order to align the granularities of two modalities, we introduce a Multi-modality Shared Dictionary (MSD) to reconstruct both visual and textual features. Besides, DGA has two important factors, i.e., the cross-modality guidance and the foreground-centric reconstruction, to facilitate the optimization of MSD. In PGU, we adopt a set of shared and learnable prototypes as the queries to extract diverse and semantically aligned features for both modalities in the granularity-unified feature space, which further promotes the ReID performance. Comprehensive experiments show that our LGUR consistently outperforms state-of-the-arts by large margins on both CUHK-PEDES and ICFG-PEDES datasets. Code will be released at https://github.com/ZhiyinShao-H/LGUR.

研究动机与目标

• 为解决由于视觉特征粒度细、文本特征粒度粗所导致的文本到图像行人重识别中显著的模态差异问题。
• 通过共享可学习词典和基于原型的机制，显式统一视觉与文本特征的粒度。
• 通过利用文本特征引导词典学习并聚焦于行人前景区域，提升跨模态对齐效果。
• 实现端到端的统一表征学习，从而提升基准数据集上的检索性能。

提出的方法

• 该框架采用基于词典的粒度对齐（DGA）模块，通过Transformer层使用多模态共享词典（MSD）重建视觉与文本特征。
• 通过在相同Transformer层中使用匹配的文本特征作为值，实现跨模态引导，以重建视觉特征，使MSD与文本描述的粒度对齐。
• 采用以前景为中心的重建策略，通过应用学习到的前景掩码抑制背景噪声，提升优化稳定性和相关性。
• 基于原型的粒度统一（PGU）模块使用共享的可学习原型作为查询，在统一空间中从两种模态中提取多样化且语义对齐的特征。
• 整个框架通过联合优化MSD与原型实现端到端训练，实现粒度统一与表征学习的联合优化。
• 该方法设计旨在不引入推理阶段计算开销的前提下减少模态差异。

Figure 1. While textual descriptions on the jackets in the all above images are the same, these jackets do in fact differ in terms of their visual details. This example well reflects the granularity gap between the two modalities, i.e. , the visual information is fine-grained while the textual featu

实验结果

研究问题

• RQ1 如何在文本到图像ReID中有效对齐细粒度视觉特征与粗粒度文本特征之间的粒度差异？
• RQ2 共享可学习词典在统一跨模态表征的同时，如何保留模态特异性语义？
• RQ3 跨模态引导与以前景为中心的重建是否能提升共享词典的学习与性能表现？
• RQ4 在统一表征空间中，跨模态共享原型如何提升特征多样性与对齐性？
• RQ5 所提出的LGUR框架在标准基准上的性能相比现有SOTA方法提升程度如何？

主要发现

• 在CUHK-PEDES数据集上，LGUR相比基线模型在Rank-1准确率上实现了6.58%的绝对提升。
• 在ICFG-PEDES数据集上，LGUR相比基线模型在Rank-1准确率上实现了6.93%的绝对提升。
• 多模态共享词典（MSD）显著优于自注意力层与非共享词典，证明其在减小粒度差异方面的有效性。
• PGU模块中共享原型的表现优于非共享原型，证实了跨模态原型共享的优势。
• 消融实验表明，DGA模块中的跨模态引导与前景掩码均为MSD学习与性能达到最优所必需。
• 定性注意力可视化显示，匹配的短语-图像块对在词典原子上的注意力得分相近，而不匹配对则不同，验证了模型统一粒度的能力。

Figure 2. Overview of the LGUR framework (shown in a), which includes a Dictionary-based Granularity Alignment (DGA) module (shown in b) and a Prototype-based Granularity Unification (PGU) module (shown in c) to achieve feature extraction that is both efficient and granularity-unified. DGA reconstru