[论文解读] Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

Nasrin Mostafazadeh, Chris Brockett+5 more|arXiv (Cornell University)|Jan 28, 2017

Multimodal Machine Learning Applications参考文献 38被引用 117

一句话总结

这篇论文介绍了 Image-Grounded Conversations (IGC) 任务，给出 IGCCrowd 数据集，并评估使用视觉和文本上下文来针对图像生成问题与回答的多模态生成与检索模型，显示出对基线的改进但仍与人类表现存在差距。

ABSTRACT

The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.

研究动机与目标

• Motivate a multimodal dialogue task where conversations are grounded in both image context and accompanying text.
• Provide a crowd-sourced, event-centric dataset (IGCCrowd) for benchmarking IGC.
• Investigate neural generation and retrieval approaches that leverage visual and textual context for Q&A and responses in IGC.
• Analyze how multimodal context affects the quality and characteristics of generated questions and responses.

提出的方法

• Define the IGC task with two steps: question generation given image I and textual context T, and response generation given I, T, and Q.
• Construct IGCCrowd and IGCTwitter datasets for training and evaluation; IGCCrowd provides 4,222 multi-turn conversations with event-centric images.
• Implement generation models that fuse visual features (VGG fc7) with textual context: V-Gen, T-Gen, and V&T-Gen (with BOW or RNN textual representations).
• Implement retrieval models using visual context only (V-Ret) or visual+textual context (V&T-Ret).
• Use beam search with reranking for decoding, combining p(h|C) with length, diversity, and V-based penalties via a scoring function for reranking.

实验结果

研究问题

• RQ1 Can multimodal (image+text) context improve natural question and response generation in image-grounded conversations?
• RQ2 How do generation and retrieval approaches compare on Q and R tasks when grounded in vision and language?
• RQ3 What dataset characteristics (event-centric grounding, frames, CaTeRS relations) reveal about IGC and its challenges?
• RQ4 To what extent do human judgments diverge from automated metrics (BLEU) in IGC settings?

主要发现

• Multimodal context improves quality of generated questions and responses compared to unimodal baselines in human evaluation.
• On multi-reference BLEU, the Visual&Textual (V&T) models outperform other models except a high-quality Visual Question Generation (VQG) baseline that benefits from event-centric training data.
• Human judges consistently prefer top-generation hypotheses over reranked ones, indicating a tradeoff between safety/genericity and content richness.
• BLEU scores are generally low due to output diversity, but V&T models achieve the best automatic performance among non-VQG baselines across test sets.
• IGCCrowd provides a robust, challenging benchmark that reveals a remaining gap between current models and human performance in multimodal dialogue tasks.