[论文解读] A Cross-Modal Distillation Network for Person Re-identification in RGB-Depth.

Frank M. Hafner, Amran Bhuiyan+2 more|arXiv (Cornell University)|Oct 27, 2018

Video Surveillance and Tracking Methods参考文献 38被引用 17

一句话总结

本文提出一种跨模态蒸馏网络，通过两步优化在RGB与深度模态之间迁移结构特征，实现特征在共享嵌入空间中的对齐。该方法在BIWI和RobotPKU数据集上实现了最先进性能，且未增加计算复杂度。

ABSTRACT

Person re-identification involves the recognition over time of individuals captured using multiple distributed sensors. With the advent of powerful deep learning methods able to learn discriminant representations for visual recognition, cross-modal person re-identification based on different sensor modalities has become viable in many challenging applications in, e.g., autonomous driving, robotics and video surveillance. Although some methods have been proposed for re-identification between infrared and RGB images, few address depth and RGB images. In addition to the challenges for each modality associated with occlusion, clutter, misalignment, and variations in pose and illumination, there is a considerable shift across modalities since data from RGB and depth images are heterogeneous. In this paper, a new cross-modal distillation network is proposed for robust person re-identification between RGB and depth sensors. Using a two-step optimization process, the proposed method transfers supervision between modalities such that similar structural features are extracted from both RGB and depth modalities, yielding a discriminative mapping to a common feature space. Our experiments investigate the influence of the dimensionality of the embedding space, compares transfer learning from depth to RGB and vice versa, and compares against other state-of-the-art cross-modal re-identification methods. Results obtained with BIWI and RobotPKU datasets indicate that the proposed method can successfully transfer descriptive structural features from the depth modality to the RGB modality. It can significantly outperform state-of-the-art conventional methods and deep neural networks for cross-modal sensing between RGB and depth, with no impact on computational complexity.

研究动机与目标

• 为解决RGB与深度传感器之间跨模态行人重识别的挑战，该挑战源于模态异质性与结构错位。
• 在外观、姿态、光照及传感器特异性噪声存在差异的情况下，实现RGB与深度模态之间的鲁棒特征迁移。
• 通过跨模态蒸馏学习一个共享的、判别性特征空间，以提升重识别准确率。
• 评估嵌入维度与特征迁移方向（深度到RGB vs. RGB到深度）在跨模态学习中的影响。
• 在使用RGB与深度数据的跨模态重识别中，超越现有最先进方法。

提出的方法

• 采用两步优化过程在RGB与深度模态之间传递监督信号，实现结构特征的对齐。
• 通过知识蒸馏将一个模态中的判别性表征迁移至另一模态，促进共享特征空间中的一致性。
• 学习一个共享嵌入空间，使来自两个模态的特征映射到一个共同的、判别性表征。
• 网络被训练以最小化RGB与深度模态间对应特征的差异，增强跨模态泛化能力。
• 该方法在提升跨数据集性能的同时，保持低计算复杂度。
• 支持双向特征迁移，允许对深度到RGB与RGB到深度的特征迁移进行评估。

实验结果

研究问题

• RQ1 嵌入空间的维度如何影响跨模态行人重识别性能？
• RQ2 哪种特征迁移方向——深度到RGB或RGB到深度——能获得更好的性能？
• RQ3 所提出的蒸馏网络能否在RGB-深度重识别中超越现有最先进方法？
• RQ4 该方法在多大程度上缓解了模态偏移，并改善了异构数据间的特征对齐？
• RQ5 该方法在实现更高准确率的同时是否保持了低计算复杂度？

主要发现

• 所提方法在BIWI和RobotPKU数据集上显著优于现有传统方法与基于深度学习的跨模态行人重识别方法。
• 从深度模态向RGB模态迁移结构特征可提升重识别准确率，证明了蒸馏过程的有效性。
• 该方法在不增加计算复杂度的前提下实现了最先进性能，保持了高效性。
• 两步优化过程成功减少了模态偏移，并增强了共享嵌入空间中的特征对齐。
• 消融实验确认嵌入维度的选择影响性能，最优设置可获得最佳结果。
• 双向迁移实验表明，深度到RGB的迁移通常优于RGB到深度，凸显了深度特征的判别能力。