[Paper Review] The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)

Andrew Shin, Yoshitaka Ushiku+1 more|arXiv (Cornell University)|Sep 21, 2016

Multimodal Machine Learning Applications25 references10 citations

TL;DR

This paper introduces FSVQA, a 1-million-sample dataset of full-sentence visual question answering derived by applying rule-based linguistic transformations to the original VQA and MS COCO caption datasets. The study establishes a baseline model using both image and question features, showing that joint visual-textual modeling significantly improves performance over unimodal baselines, with FSVQA accuracy reaching 64.3% on the validation set.

ABSTRACT

Visual Question Answering (VQA) task has showcased a new stage of interaction between language and vision, two of the most pivotal components of artificial intelligence. However, it has mostly focused on generating short and repetitive answers, mostly single words, which fall short of rich linguistic capabilities of humans. We introduce Full-Sentence Visual Question Answering (FSVQA) dataset, consisting of nearly 1 million pairs of questions and full-sentence answers for images, built by applying a number of rule-based natural language processing techniques to original VQA dataset and captions in the MS COCO dataset. This poses many additional complexities to conventional VQA task, and we provide a baseline for approaching and evaluating the task, on top of which we invite the research community to build further improvements.

Motivation & Objective

• To address the limitation of existing VQA datasets that focus on short, single-word answers, which fail to capture the full expressiveness of human language.
• To propose a large-scale, publicly available dataset of full-sentence answers for visual question answering, enabling more natural and complex language generation.
• To establish a baseline for FSVQA using multimodal features and evaluate performance with both standard NLP metrics and context-aware accuracy measures.
• To encourage the research community to develop more effective models for generating linguistically rich, contextually accurate answers in visual question answering.

Proposed method

• Leveraged rule-based natural language processing to convert original VQA questions and answers into full-sentence answers using tense, subject-verb agreement, and modal verb transformations.
• Applied linguistic rules to convert MS COCO captions into question-answer pairs, expanding the dataset with diverse, natural-language queries.
• Constructed a dataset of nearly 1 million image-question-answer triplets by combining transformed VQA data and caption-derived QA pairs.
• Trained a multimodal model using both image features (from ResNet) and question features (from LSTM) to generate full-sentence answers.
• Evaluated models using standard captioning metrics (BLEU, METEOR, CIDEr) and introduced two novel metrics: VQA accuracy (presence of original short answer) and FSVQA accuracy (sentence-level match with ground truth).
• Used a hybrid evaluation strategy: VQA accuracy on original VQA subset, FSVQA accuracy on the full FSVQA dataset, to ensure contextual correctness.

Experimental results

Research questions

• RQ1 Can rule-based linguistic transformation effectively generate diverse, grammatically correct full-sentence answers from existing VQA and caption data?
• RQ2 How does the performance of multimodal models vary when using only image features, only question features, or both features for full-sentence VQA?
• RQ3 To what extent do standard NLP evaluation metrics like BLEU and CIDER correlate with human-annotated answer correctness in full-sentence VQA?
• RQ4 Can VQA accuracy (presence of original short answer) and FSVQA accuracy (sentence-level match) serve as reliable and complementary evaluation metrics for full-sentence VQA?
• RQ5 What is the impact of increasing answer diversity—especially from 1000 to 40,000 unique answers—on model generalization and performance?

Key findings

• The FSVQA dataset contains nearly 1 million image-question-answer triplets, with approximately 40 times more unique answers than the original VQA dataset.
• The baseline model using both image and question features achieved an FSVQA accuracy of 64.3% on the validation set, significantly outperforming unimodal baselines.
• Models using only question features performed comparably to multimodal models, suggesting that the task reduces to semantic QA when visual context is ignored.
• Models using only image features performed poorly, achieving at best 1/k accuracy per image, where k is the number of questions per image, due to lack of question-specific grounding.
• VQA accuracy and FSVQA accuracy showed complementary strengths: VQA accuracy detected correct semantic content, while FSVQA accuracy ensured contextual and syntactic correctness.
• The results indicate that full-sentence VQA is a significantly more complex task than standard VQA, requiring joint visual and linguistic understanding beyond simple classification.