[论文解读] Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations

Fenglin Liu, Yuanxin Liu+3 more|arXiv (Cornell University)|May 15, 2019

Multimodal Machine Learning Applications参考文献 43被引用 45

一句话总结

本文提出互迭代注意力（MIA），将视觉区域与文本概念对齐，产生语义支撑的图像表征，在多种基线下提升图像描述和 VQA 表现。

ABSTRACT

In vision-and-language grounding problems, fine-grained representations of the image are considered to be of paramount importance. Most of the current systems incorporate visual features and textual concepts as a sketch of an image. However, plainly inferred representations are usually undesirable in that they are composed of separate components, the relations of which are elusive. In this work, we aim at representing an image with a set of integrated visual regions and corresponding textual concepts, reflecting certain semantics. To this end, we build the Mutual Iterative Attention (MIA) module, which integrates correlated visual features and textual concepts, respectively, by aligning the two modalities. We evaluate the proposed approach on two representative vision-and-language grounding tasks, i.e., image captioning and visual question answering. In both tasks, the semantic-grounded image representations consistently boost the performance of the baseline models under all metrics across the board. The results demonstrate that our approach is effective and generalizes well to a wide range of models for image-related applications. (The code is available at https://github.com/fenglinliu98/MIA)

研究动机与目标

• Motivate the need for integrated image representations that jointly reflect visual regions and textual concepts.
• Propose a mechanism (MIA) to iteratively align and integrate multi-modal features without supervision.
• Demonstrate generality by improving baseline models on image captioning and VQA datasets.
• Show that semantic-grounded representations are robust across model architectures and feature types.

提出的方法

• Represent images with paired visual features (grid or RoI) and textual concepts (visual words).
• Use Mutual Attention to align features across domains, with a multi-head attention mechanism and feed-forward refinements.
• Iteratively apply mutual attention (N iterations) with shared parameters to produce I_N and T_N, then combine as MIA(I,T)=LayerNorm(I_N+T_N).
• Adopt a distantly supervised training regime by integrating MIA into downstream tasks (captioning and VQA) without requiring aligned supervision.
• Provide implementation details such as 8 attention heads (k=8) and 2 iterations (N=2) for best validation performance.

实验结果

研究问题

• RQ1 Can an iterative cross-modal alignment (MIA) produce semantically grounded image representations that improve downstream vision-language tasks?
• RQ2 Do integrated representations outperform traditional single-domain features across image captioning and VQA baselines?
• RQ3 How does the number of iterations affect alignment quality and task performance?
• RQ4 Is the improvement due to semantic grounding rather than simply adding more features from another modality?
• RQ5 Does MIA generalize across different visual features (grid vs RoI) and textual concept sets?

主要发现

• MIA consistently improves baselines on image captioning (SPICE and CIDEr gains) and VQA accuracy.
• Using integrated representations enables baselines to attend to semantically grounded feature collections rather than separate features.
• MIA achieves gains across both RNN-based (Up-Down) and self-attention (Transformer) captioning models, and improves BAN/Up-Down on VQA v2.0.
• Ablation shows improvements even with single-modal inputs refined by MIA, and larger gains when both I_N and T_N are combined.
• Iteration analysis shows best performance around N=2; too many iterations can over-concentrate and reduce information.