[论文解读] Order embeddings and character-level convolutions for multimodal alignment

Jônatas Wehrmann, Anderson Mattjie+1 more|arXiv (Cornell University)|Jun 3, 2017

Multimodal Machine Learning Applications参考文献 33被引用 21

一句话总结

本文提出一种用于图像-文本对齐的字符级卷积神经网络，用原始字符卷积替代词嵌入和RNN，实现更快、更简单的训练，且参数更少。通过使用顺序嵌入保留语义层次结构，并优化对比损失，该方法在Microsoft COCO数据集上达到最先进性能。

ABSTRACT

With the novel and fast advances in the area of deep neural networks, several challenging image-based tasks have been recently approached by researchers in pattern recognition and computer vision. In this paper, we address one of these tasks, which is to match image content with natural language descriptions, sometimes referred as multimodal content retrieval. Such a task is particularly challenging considering that we must find a semantic correspondence between captions and the respective image, a challenge for both computer vision and natural language processing areas. For such, we propose a novel multimodal approach based solely on convolutional neural networks for aligning images with their captions by directly convolving raw characters. Our proposed character-based textual embeddings allow the replacement of both word-embeddings and recurrent neural networks for text understanding, saving processing time and requiring fewer learnable parameters. Our method is based on the idea of projecting both visual and textual information into a common embedding space. For training such embeddings we optimize a contrastive loss function that is computed to minimize order-violations between images and their respective descriptions. We achieve state-of-the-art performance in the largest and most well-known image-text alignment dataset, namely Microsoft COCO, with a method that is conceptually much simpler and that possesses considerably fewer parameters than current approaches.

研究动机与目标

• 解决多模态检索中图像与自然语言描述对齐的挑战。
• 消除对预训练词嵌入和RNN的依赖，后者计算成本高且内存占用大。
• 简化文本理解架构，同时保持高性能。
• 提升低资源或多语言NLP场景下的效率与可扩展性。

提出的方法

• 使用一维卷积层直接处理原始字符序列，替代词嵌入和RNN。
• 应用带填充的卷积操作，结合可学习滤波器，生成字符级文本嵌入。
• 采用顺序嵌入建模图像字幕层次结构中的偏序关系。
• 优化对比损失函数，惩罚正样本对（图像-字幕）之间的顺序违规。
• 将视觉与文本特征映射到共享嵌入空间，实现跨模态对齐。
• 在COCO数据集上端到端训练模型，无需预训练。

实验结果

研究问题

• RQ1 原始字符级卷积能否在图像-文本对齐任务中有效替代词嵌入和RNN？
• RQ2 使用顺序嵌入是否能通过保留字幕中的语义层次结构来提升性能？
• RQ3 更简单、参数更少的架构能否在图像-文本检索任务中超越复杂的最先进模型？
• RQ4 与基于RNN的基线方法相比，该方法在训练效率和推理速度方面表现如何？

主要发现

• 所提方法在Microsoft COCO数据集上的图像-文本检索任务中达到最先进性能。
• 与现有基于RNN和词嵌入的方法相比，该模型显著减少了可学习参数数量。
• 该方法训练更快、更简单，无需预训练嵌入或复杂的序列建模。
• 失败案例揭示了在复杂场景中处理罕见或模糊视觉概念的挑战。
• 消融实验表明，仅使用字符级卷积已足够实现优异性能，在某些设置下优于词嵌入基线。
• 顺序嵌入的使用有助于更好地对齐层次化字幕结构，从而提升检索准确率。