[论文解读] MHTN: Modal-adversarial Hybrid Transfer Network for Cross-modal Retrieval

Xin Huang, Yuxin Peng+1 more|arXiv (Cornell University)|Aug 8, 2017

Multimodal Machine Learning Applications参考文献 40被引用 9

一句话总结

本文提出MHTN，一种新颖的端到端模态对抗性混合迁移网络，将大规模单模态数据集（如ImageNet）中的知识迁移至跨模态检索任务，提升文本、图像、视频、音频和3D数据之间的语义一致性。通过结合星型结构的知识迁移子网络与模态对抗性训练机制，MHTN增强了跨模态表征学习，在四个基准数据集上实现了最先进性能，包括具有挑战性的五模态XMedia数据集。

ABSTRACT

Cross-modal retrieval has drawn wide interest for retrieval across different modalities of data. However, existing methods based on DNN face the challenge of insufficient cross-modal training data, which limits the training effectiveness and easily leads to overfitting. Transfer learning is for relieving the problem of insufficient training data, but it mainly focuses on knowledge transfer only from large-scale datasets as single-modal source domain to single-modal target domain. Such large-scale single-modal datasets also contain rich modal-independent semantic knowledge that can be shared across different modalities. Besides, large-scale cross-modal datasets are very labor-consuming to collect and label, so it is significant to fully exploit the knowledge in single-modal datasets for boosting cross-modal retrieval. This paper proposes modal-adversarial hybrid transfer network (MHTN), which to the best of our knowledge is the first work to realize knowledge transfer from single-modal source domain to cross-modal target domain, and learn cross-modal common representation. It is an end-to-end architecture with two subnetworks: (1) Modal-sharing knowledge transfer subnetwork is proposed to jointly transfer knowledge from a large-scale single-modal dataset in source domain to all modalities in target domain with a star network structure, which distills modal-independent supplementary knowledge for promoting cross-modal common representation learning. (2) Modal-adversarial semantic learning subnetwork is proposed to construct an adversarial training mechanism between common representation generator and modality discriminator, making the common representation discriminative for semantics but indiscriminative for modalities to enhance cross-modal semantic consistency during transfer process. Comprehensive experiments on 4 widely-used datasets show its effectiveness and generality.

研究动机与目标

• 为解决深度跨模态检索中跨模态训练数据标注不足的挑战。
• 利用大规模单模态数据集（如ImageNet）中的丰富、模态无关语义知识，以增强跨模态学习。
• 在从单模态源域到多模态目标域的知识迁移过程中，保持跨模态语义一致性。
• 开发一个端到端框架，联合优化知识迁移与模态间语义对齐。
• 在包括五模态检索在内的多样化跨模态检索任务中，展示方法的泛化能力与有效性。

提出的方法

• 一种星型结构的模态共享知识迁移子网络，联合将单模态源域（如ImageNet）的知识迁移至目标域的所有模态，蒸馏出模态无关的语义知识。
• 一种模态对抗性语义学习子网络，通过在公共表征生成器与模态判别器之间引入对抗性训练，使共享表征在语义上具有判别性，但对模态保持不变。
• 对抗性训练显式减小了异质性差距，通过促使公共表征在不同模态间难以区分，同时保持语义判别能力。
• 该框架整合了源域监督损失，即使在源域与目标域类别不匹配的情况下，也能利用源域的通用知识。
• 模型采用混合损失函数进行端到端训练，结合了跨模态对比损失、对抗性损失和源域监督损失。
• 该架构在特征提取器上分别采用AlexNet和VGG19进行评估，证明了其对主干网络选择的鲁棒性。

实验结果

研究问题

• RQ1 能否有效从大规模单模态数据集中迁移知识，以提升跨模态检索性能？
• RQ2 如何从单模态源域中蒸馏出模态无关的语义知识，以增强跨模态表征学习？
• RQ3 公共表征生成器与模态判别器之间的对抗性训练，在多大程度上提升了跨模态语义一致性？
• RQ4 在类别不匹配的情况下，源域监督是否仍能为跨模态检索带来收益？
• RQ5 所提出的MHTN框架是否能在包括五模态在内的多样化跨模态检索任务中实现泛化？

主要发现

• MHTN在四个广泛使用的跨模态检索数据集上实现了最先进性能，包括具有挑战性的五模态XMedia数据集。
• 移除源图像路径（MHTN (NoSource)）导致性能持续下降，证实了单模态源知识在跨模态学习中的价值。
• 移除模态对抗性语义学习子网络（MHTN (NoSLnet)）导致检索准确率降低，证明其在保持跨模态语义一致性方面的作用。
• 移除对抗性训练策略（MHTN (NoAdver)）导致性能下降，表明模态不变表征学习有助于提升跨模态对齐。
• 在源路径与目标路径中均将AlexNet替换为VGG19后，所有数据集的检索准确率均有所提升，尤其在Pascal Sentences数据集上表现更优，表明该方法对主干网络选择具有鲁棒性。
• 即使在无图像的跨模态任务中（如Text → Video、3D → Text），MHTN仍能从源图像路径中显著获益，证明其在模态间具有强大的泛化能力。