[论文解读] Learning Semantic Concepts and Order for Image and Sentence Matching

Yan Huang, Qi Wu+1 more|arXiv (Cornell University)|Dec 6, 2017

Multimodal Machine Learning Applications参考文献 28被引用 25

一句话总结

本文提出了一种语义增强的图像与句子匹配模型，通过联合学习高层语义概念（例如物体、属性、动作）及其正确的语义顺序，提升了图像表征能力。采用多区域多标签CNN进行概念检测，以及基于上下文门控的句子生成方案进行顺序学习，该模型在MSCOCO和Flickr30k基准上实现了最先进性能，分别达到42.8%和33.1%的top-1图像检索准确率。

ABSTRACT

Image and sentence matching has made great progress recently, but it remains challenging due to the large visual-semantic discrepancy. This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence. In this work, we propose a semantic-enhanced image and sentence matching model, which can improve the image representation by learning semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its semantic concepts, including objects, properties, actions, etc. Then, considering that different orders of semantic concepts lead to diverse semantic meanings, we use a context-gated sentence generation scheme for semantic order learning. It simultaneously uses the image global context containing concept relations as reference and the groundtruth semantic order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets.

研究动机与目标

• 通过引入高层语义概念来增强图像表征，以解决图像与句子匹配中的视觉-语义差异问题。
• 建模概念的正确语义顺序，这对准确匹配至关重要，但在现有方法中常被忽略。
• 联合学习图像-句子匹配与句子生成任务，实现端到端优化。
• 通过基于区域的特征提取捕获前景与背景概念，提升细粒度匹配能力。

提出的方法

• 使用多区域多标签CNN从图像的多个区域预测语义概念（物体、属性、动作），实现全面的概念检测。
• 通过门控融合模块将预测的语义概念与全局图像上下文（空间关系）结合，形成上下文感知的图像表征。
• 基于真实句子顺序作为监督信号，采用上下文门控的句子生成模块学习概念的正确语义顺序。
• 通过结构化匹配目标与句子生成目标，联合优化图像与句子表征，实现端到端学习。
• 通过整合语义概念及其有序结构，增强图像表征，提升与自然语言描述的一致性。

实验结果

研究问题

• RQ1 如何改进图像表征以更好地捕捉超越像素级特征的高层语义概念？
• RQ2 语义顺序在减少图像-句子匹配中的视觉-语义差异方面起到什么作用？
• RQ3 联合学习图像-句子匹配与句子生成是否能提升表征质量？
• RQ4 所提出的上下文门控生成方案在从图像上下文与真实句子中学习正确语义顺序方面效果如何？

主要发现

• 所提模型在MSCOCO数据集上达到42.8%的top-1图像检索准确率，优于先前最先进方法。
• 在Flickr30k数据集上，模型实现33.1%的top-1检索准确率，显著优于现有方法。
• 消融实验表明，同时引入语义概念与顺序学习可提升句子检索与标注性能，完整模型优于各类消融变体。
• 在MSCOCO数据集上，模型实现40.2%的图像标注准确率，显著高于先前方法如VSE++（32.9%）与OEM（23.3%）。
• 使用VGGNet进行概念检测带来的性能增益高于ResNet，表明概念检测中特征质量对最终性能有显著影响。