[论文解读] SCH-GAN: Semi-supervised Cross-modal Hashing by Generative Adversarial Network

Jian Zhang, Yuxin Peng+1 more|arXiv (Cornell University)|Feb 7, 2018

Advanced Image and Video Retrieval Techniques参考文献 40被引用 4

一句话总结

该论文提出SCH-GAN，一种基于生成对抗网络的半监督跨模态哈希方法，通过从无标签数据中生成难样本、边界正样本以利用无标签数据，从而提升跨模态检索性能。生成模型对抗性地选择有信息量的样本，而判别模型则学习区分这些样本，使NUSWIDE、Wikipedia和MIRFlickr数据集上的平均精度均值（MAP）达到最高0.758，性能达到当前最优水平。

ABSTRACT

Cross-modal hashing aims to map heterogeneous multimedia data into a common Hamming space, which can realize fast and flexible retrieval across different modalities. Supervised cross-modal hashing methods have achieved considerable progress by incorporating semantic side information. However, they mainly have two limitations: (1) Heavily rely on large-scale labeled cross-modal training data which are labor intensive and hard to obtain. (2) Ignore the rich information contained in the large amount of unlabeled data across different modalities, especially the margin examples that are easily to be incorrectly retrieved, which can help to model the correlations. To address these problems, in this paper we propose a novel Semi-supervised Cross-Modal Hashing approach by Generative Adversarial Network (SCH-GAN). We aim to take advantage of GAN's ability for modeling data distributions to promote cross-modal hashing learning in an adversarial way. The main contributions can be summarized as follows: (1) We propose a novel generative adversarial network for cross-modal hashing. In our proposed SCH-GAN, the generative model tries to select margin examples of one modality from unlabeled data when giving a query of another modality. While the discriminative model tries to distinguish the selected examples and true positive examples of the query. These two models play a minimax game so that the generative model can promote the hashing performance of discriminative model. (2) We propose a reinforcement learning based algorithm to drive the training of proposed SCH-GAN. The generative model takes the correlation score predicted by discriminative model as a reward, and tries to select the examples close to the margin to promote discriminative model by maximizing the margin between positive and negative data. Experiments on 3 widely-used datasets verify the effectiveness of our proposed approach.

研究动机与目标

• 为解决监督型跨模态哈希方法严重依赖昂贵标注数据且未能充分利用无标签数据的局限性。
• 利用大规模无标签数据中的丰富结构信息，特别是来自另一模态的边界样本，以提升哈希性能。
• 设计一种半监督框架，通过在难且有信息量的样本上对抗性训练生成模型，增强判别模型的学习能力。
• 将强化学习整合到GAN训练过程中，其中生成模型根据判别模型的关联度分数获得奖励。

提出的方法

• 设计一种新颖的基于GAN的框架，当接收来自某一模态的查询时，生成器从无标签数据中选择边界正样本。
• 判别器评估查询与生成样本之间的相关性，将其与真实正样本区分开来。
• 两个模型通过极小化极大博弈进行训练：生成器不断改进其样本选择策略以挑战判别器，从而增强判别模型的鲁棒性。
• 强化学习驱动生成器的训练过程：奖励信号来自判别器预测的相关度分数，鼓励生成器选择靠近分类边界的样本。
• 训练过程中使用三元组排序损失优化判别模型，确保正样本在汉明空间中比负样本更接近。
• 该框架通过端到端的对抗性与对比性目标联合训练，实现有效的跨模态嵌入学习。

实验结果

研究问题

• RQ1 生成对抗网络能否有效利用无标签数据以提升跨模态哈希性能？
• RQ2 在无标签数据中使用边界样本进行对抗性训练，能否增强判别模型区分正负样本对的能力？
• RQ3 将强化学习与判别器反馈结合，能否提升跨模态哈希中生成的难样本的质量？
• RQ4 所提出的SCH-GAN在检索准确率方面与当前最先进的监督型和无监督型跨模态哈希方法相比表现如何？

主要发现

• 在NUSWIDE数据集上，SCH-GAN将图像到文本检索的平均MAP从0.505提升至0.730，文本到图像检索的平均MAP从0.490提升至0.758，优于最佳无监督方法CCQ。
• SCH-GAN优于最佳监督方法SePH，在图像到文本检索中平均MAP提升0.015（从0.715提升至0.730），在文本到图像检索中提升0.104（从0.654提升至0.758）。
• 与基于深度学习的方法DCMH相比，SCH-GAN在图像到文本检索中将MAP从0.652提升至0.730，在文本到图像检索中从0.696提升至0.758。
• 消融实验表明，对抗性训练显著提升性能，SCH-GAN在所有三个数据集上均持续优于基线方法Dis（无对抗训练）。
• 精确率-召回率曲线与Top-K精确率结果表明，SCH-GAN在所有三个数据集上均达到最高准确率，验证了其鲁棒性与有效性。
• 训练过程中，判别模型的准确率在每次生成器更新后均有所提升，表明生成器成功选择了有助于提升判别模型泛化能力的信息量大、难样本。