[论文解读] Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

Kai Niu, Yan Huang+2 more|arXiv (Cornell University)|Jun 23, 2019

Video Surveillance and Tracking Methods参考文献 49被引用 12

一句话总结

本文提出了一种多粒度图像-文本对齐（MIA）模型，用于基于描述的行人重识别，通过在全局-全局、全局-局部和局部-局部三个层次上分层对齐图像与文本，解决跨模态相似性与细粒度区分问题。该方法采用端到端可训练的框架与分步训练策略，在CUHK-PEDES数据集上实现了最先进性能。

ABSTRACT

Description-based person re-identification (Re-id) is an important task in video surveillance that requires discriminative cross-modal representations to distinguish different people. It is difficult to directly measure the similarity between images and descriptions due to the modality heterogeneity (the cross-modal problem). And all samples belonging to a single category (the fine-grained problem) makes this task even harder than the conventional image-description matching task. In this paper, we propose a Multi-granularity Image-text Alignments (MIA) model to alleviate the cross-modal fine-grained problem for better similarity evaluation in description-based person Re-id. Specifically, three different granularities, i.e., global-global, global-local and local-local alignments are carried out hierarchically. Firstly, the global-global alignment in the Global Contrast (GC) module is for matching the global contexts of images and descriptions. Secondly, the global-local alignment employs the potential relations between local components and global contexts to highlight the distinguishable components while eliminating the uninvolved ones adaptively in the Relation-guided Global-local Alignment (RGA) module. Thirdly, as for the local-local alignment, we match visual human parts with noun phrases in the Bi-directional Fine-grained Matching (BFM) module. The whole network combining multiple granularities can be end-to-end trained without complex pre-processing. To address the difficulties in training the combination of multiple granularities, an effective step training strategy is proposed to train these granularities step-by-step. Extensive experiments and analysis have shown that our method obtains the state-of-the-art performance on the CUHK-PEDES dataset and outperforms the previous methods by a significant margin.

研究动机与目标

• 为解决基于描述的行人重识别中的跨模态细粒度问题，即行人的图像视觉相似且描述语义重叠。
• 通过在不同粒度级别上实现多级对齐，提升图像与文本描述之间的跨模态相似性度量。
• 设计一种端到端可训练的模型，避免复杂的预处理以及姿态或部件标签等外部标注。
• 开发一种分步训练策略，按顺序有效训练多个粒度级别，以稳定学习过程并提升性能。

提出的方法

• 全局对比（GC）模块通过匹配整体图像与描述的表征，实现全局-全局对齐，以捕捉全局上下文信息。
• 关系引导的全局-局部对齐（RGA）模块通过建模局部部件与全局上下文之间的关系，自适应地突出可区分的局部组件，同时抑制无关区域。
• 双向细粒度匹配（BFM）模块通过将视觉人体部件与描述中的名词短语进行匹配，实现局部-局部对齐。
• 采用分步训练策略，按顺序依次训练三个粒度级别——先GC，再RGA，最后BFM——以稳定优化过程并提升收敛性。
• 整个MIA模型为端到端可训练，无需外部线索、部件标注或复杂预处理。

实验结果

研究问题

• RQ1 多粒度对齐如何提升基于描述的行人重识别中的跨模态相似性？
• RQ2 在全局、全局-局部与局部-局部层级上的分层对齐是否能增强细粒度行人图像的区分能力？
• RQ3 分步训练策略如何提升多粒度图像-文本对齐的训练稳定性和性能？
• RQ4 超参数对MIA模型性能有何影响？

主要发现

• MIA模型在CUHK-PEDES数据集上实现了最先进性能，显著优于先前方法。
• 当超参数λ₁=1.5且λ₂=0.7时，R@1准确率达到47.8%，总检索得分为197.2。
• 性能在λ₁=1.5与λ₂=0.7时达到峰值，表明全局与局部对齐监督之间达到最佳平衡。
• 失败案例主要源于描述覆盖不全或属性模糊不清，如未提及的配饰或颜色模糊性。
• 消融实验证实，GC、RGA与BFM各组件均对性能提升有贡献，完整MIA模型达到最高检索得分。