[Paper Review] Visual Madlibs: Fill in the blank Image Generation and Question Answering

Licheng Yu, Eunbyung Park+2 more|arXiv (Cornell University)|May 31, 2015

Multimodal Machine Learning ApplicationsComputer Science32 references80 citations

TL;DR

This paper introduces the Visual Madlibs dataset, a collection of 360,001 targeted, fill-in-the-blank descriptions for 10,738 images, using automated templates to elicit detailed descriptions about people, objects, actions, and contextual elements. It proposes two new tasks—focused description generation and multiple-choice image question answering—and evaluates joint-embedding and CNN+LSTM models, showing that the latter achieve higher BLEU scores on generation while the former outperforms on fine-grained multiple-choice QA, especially with hard negative examples.

ABSTRACT

In this paper, we introduce a new dataset consisting of 360,001 focused natural language descriptions for 10,738 images. This dataset, the Visual Madlibs dataset, is collected using automatically produced fill-in-the-blank templates designed to gather targeted descriptions about: people and objects, their appearances, activities, and interactions, as well as inferences about the general scene or its broader context. We provide several analyses of the Visual Madlibs dataset and demonstrate its applicability to two new description generation tasks: focused description generation, and multiple-choice question-answering for images. Experiments using joint-embedding and deep learning methods show promising results on these tasks.

Motivation & Objective

• To develop a scalable, automated method for collecting detailed, targeted natural language descriptions of images using fill-in-the-blank templates.
• To create a new dataset, Visual Madlibs, with 360,001 descriptions across 12 question types focusing on appearances, actions, interactions, and contextual inferences.
• To define and evaluate two new tasks: focused description generation and multiple-choice image question answering with nuanced difficulty control.
• To compare the performance of joint-embedding models and end-to-end CNN+LSTM sequence models on these new tasks.
• To provide publicly released data and benchmarks to advance research in grounded language understanding and visual captioning.

Proposed method

• Automatically generate 12 types of fill-in-the-blank templates targeting specific visual attributes: e.g., 'The person is [blank]' for actions, 'The frisbee is [blank]' for appearance.
• Collect 360,001 descriptions from human annotators using these templates on 10,738 images, ensuring focused, targeted linguistic annotations.
• Train a CNN+LSTM model per question type to generate descriptions by conditioning on image features and the prompt prefix (e.g., 'The chair is').
• Use a joint-embedding model (nCCA) to map images and text into a shared space, enabling zero-shot description generation and multiple-choice answer selection via cosine similarity.
• For multiple-choice QA, compute Word2Vec cosine similarity between generated descriptions and candidate answers, selecting the most similar as the predicted answer.
• Apply bounding box features from R-CNN detectors to improve attribute prediction, comparing performance with ground-truth and detected boxes.

Experimental results

Research questions

• RQ1 Can automated fill-in-the-blank templates effectively elicit detailed, targeted descriptions of visual content, including actions, appearances, and contextual inferences?
• RQ2 How do joint-embedding models and end-to-end sequence models compare in generating focused image descriptions across diverse question types?
• RQ3 Can multiple-choice question answering with controlled negative examples provide a nuanced, reliable evaluation metric for image description systems?
• RQ4 Does using detected bounding boxes for visual features improve performance on attribute-related description tasks?
• RQ5 How do human-annotated descriptions in Visual Madlibs compare in detail and diversity to generic image captions in MS COCO?

Key findings

• The CNN+LSTM model achieved higher BLEU-1 and BLEU-2 scores (e.g., 0.71 for 'obj pos' and 0.64 for 'per loc') compared to nCCA on focused description generation, indicating superior fluency and accuracy.
• The nCCA model outperformed CNN+LSTM on multiple-choice question answering, especially on the hard version, achieving 0.52 accuracy on 'scene' and 0.56 on 'obj aff' with the full model.
• Using detected bounding boxes improved nCCA accuracy on attribute prediction (e.g., 0.60 for 'obj aff' vs. 0.56 with full image), but not on other question types.
• The human-annotated subset of hard multiple-choice questions (with at least 3/5 Turkers agreeing on the correct answer) showed consistent performance patterns, with nCCA maintaining higher accuracy than CNN+LSTM on this filtered set.
• The Visual Madlibs dataset contains more detailed and contextually rich descriptions than MS COCO, with 12 distinct question types covering actions, appearances, emotions, and temporal inferences.
• The nCCA model trained on all question types together (nCCA(all)) achieved higher accuracy on easy multiple-choice questions but underperformed on fine-grained hard versions, suggesting task-specific fine-tuning may be beneficial.