[论文解读] Deep Semantic Multimodal Hashing Network for Scalable Multimedia Retrieval.

Lu Jin, Jinhui Tang+3 more|arXiv (Cornell University)|Jan 9, 2019

Advanced Image and Video Retrieval Techniques参考文献 75被引用 5

一句话总结

本文提出深度语义多模态哈希网络（DSMHN），这是一种统一的深度学习框架，通过保留模态间相似性、模态内语义标签以及比特平衡约束，联合学习特定模态的哈希函数。通过在训练过程中直接将语义标签嵌入哈希码，DSMHN 在三个基准数据集上的检索性能显著优于现有最先进方法。

ABSTRACT

Hashing has been widely applied to multimodal retrieval on large-scale multimedia data due to its efficiency in computation and storage. Particularly, deep hashing has received unprecedented research attention in recent years, owing to its perfect retrieval performance. However, most of existing deep hashing methods learn binary hash codes by preserving the similarity relationship while without exploiting the semantic labels of data points, which result in suboptimal binary codes. In this work, we propose a novel Deep Semantic Multimodal Hashing Network for scalable multimodal retrieval. In DSMHN, two sets of modality-specific hash functions are jointly learned by explicitly preserving both the inter-modality similarities and the intra-modality semantic labels. Specifically, with the assumption that the learned hash codes should be optimal for task-specific classification, two stream networks are jointly trained to learn the hash functions by embedding the semantic labels on the resultant hash codes. Different from previous deep hashing methods, which are tied to some particular forms of loss functions, the proposed deep hashing framework can be flexibly integrated with different types of loss functions. In addition, the bit balance property is investigated to generate binary codes with each bit having 50% probability to be 1 or -1. Moreover, a unified deep multimodal hashing framework is proposed to learn compact and high-quality hash codes by exploiting the feature representation learning, inter-modality similarity preserving learning, semantic label preserving learning and hash functions learning with bit balanced constraint simultaneously. We conduct extensive experiments for both unimodal and cross-modal retrieval tasks on three widely-used multimodal retrieval datasets. The experimental result demonstrates that DSMHN significantly outperforms state-of-the-art methods.

研究动机与目标

• 为解决现有深度哈希方法在二进制码学习过程中忽略语义标签的局限性。
• 通过在哈希码学习过程中显式整合语义标签信息，提升检索准确性。
• 开发一种灵活的深度哈希框架，兼容多种损失函数，超越固定架构的限制。
• 确保二进制码的平衡性，使每位比特有约 50% 的概率为 1 或 -1，从而提升码的质量。
• 将特征表示学习、相似性保留、语义标签学习与比特平衡统一为一个端到端的联合框架。

提出的方法

• 通过共享特征和模态特定特征，联合训练两个分支网络，以学习特定模态的哈希函数。
• 在训练过程中直接将语义标签嵌入哈希码，以指导学习具有语义意义的二进制码。
• 通过最小化语义相似的跨模态样本对之间哈希码的距离，保留模态间相似性。
• 通过将哈希码与其对应的语义嵌入对齐，强制实现模态内语义标签的保留。
• 施加比特平衡约束，确保哈希码中每位比特有大致相等的概率为 1 或 -1。
• 该框架设计灵活，兼容多种损失函数，支持与多样化优化目标的集成。

实验结果

研究问题

• RQ1 在深度哈希码学习中显式整合语义标签是否能提升多模态检索任务的检索性能？
• RQ2 联合学习模态间相似性与模态内语义标签对所学习二进制码的质量有何影响？
• RQ3 强制实施比特平衡在多大程度上能提升深度多模态哈希模型的性能？
• RQ4 所提出的框架能否灵活扩展至不同损失函数，而不会损害性能？
• RQ5 在单模态与跨模态检索场景下，DSMHN 与最先进方法相比表现如何？

主要发现

• DSMHN 在三个广泛使用的多模态检索数据集上显著优于现有最先进深度哈希方法。
• 将语义标签整合到哈希码学习过程中，可生成更具区分性与语义意义的二进制码。
• 比特平衡约束有助于提升泛化能力，并使哈希码分布更加均匀。
• 联合优化特征学习、相似性保留与语义标签对齐，显著提升了单模态与跨模态任务的检索准确性。
• 所提框架展现出强大的泛化能力与灵活性，可无缝集成多种损失函数，且性能无下降。