[Paper Review] VQA: Visual Question Answering

Aishwarya Agrawal, Jiasen Lu+5 more|arXiv (Cornell University)|May 3, 2015

Multimodal Machine Learning Applications41 references1,094 citations

TL;DR

This paper introduces Visual Question Answering (VQA), a task that requires models to answer open-ended, natural language questions about images using multimodal understanding. It presents a large-scale dataset with ~250K images, ~760K questions, and ~10M answers, enabling automatic evaluation and benchmarking of models that combine vision, language, and reasoning, with state-of-the-art methods achieving significant performance gains over baselines.

ABSTRACT

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

Motivation & Objective

• To propose Visual Question Answering (VQA) as a new, challenging benchmark for multimodal AI that requires both visual understanding and language reasoning.
• To create a large-scale, diverse dataset of ~250K images, ~760K questions, and ~10M answers to support research in multimodal understanding.
• To enable automatic, quantitative evaluation of VQA models through open-ended and multiple-choice answer formats.
• To establish a foundation for evaluating models that integrate computer vision, natural language processing, and knowledge reasoning in a unified task.
• To foster progress in AI by providing a task that is both accessible and challenging, with a clear path for tracking performance improvements.

Proposed method

• The VQA task is defined as generating a free-form natural language answer to a question about a given image, requiring integration of visual and linguistic understanding.
• A large-scale dataset is constructed using MS COCO images and a new abstract scene dataset, with three questions per image collected via Amazon Mechanical Turk.
• Each question is answered by ten human annotators, ensuring diverse and reliable ground-truth answers for evaluation.
• The dataset supports two evaluation formats: open-ended (free-form answers) and multiple-choice (selection from predefined options), enabling both generative and classification-based evaluation.
• Baselines are developed using state-of-the-art visual features (e.g., from deep convolutional networks) combined with recurrent neural networks for question encoding and answer generation.
• An annual challenge and workshop are organized to track progress and promote innovation in VQA research.

Experimental results

Research questions

• RQ1 What types of visual and linguistic reasoning are required to answer open-ended questions about images, and how do they vary across question categories?
• RQ2 How well can current models perform on VQA compared to human performance, and what are the key failure modes?
• RQ3 To what extent does the quality of visual features and language modeling affect VQA performance?
• RQ4 How does the inclusion of commonsense and world knowledge impact answer accuracy in VQA?
• RQ5 Can automatic evaluation metrics reliably track progress in VQA, especially for open-ended answers?

Key findings

• The VQA dataset contains approximately 250,000 images, 760,000 questions, and 10 million answers, making it one of the largest and most diverse benchmarks for multimodal understanding.
• The dataset includes both real-world images from MS COCO and abstract scenes designed to isolate high-level reasoning from visual complexity.
• Multiple baselines using deep visual features and recurrent language models show measurable performance, though significant gaps remain compared to human performance.
• The task is amenable to automatic evaluation, particularly through multiple-choice formats, enabling reliable and scalable benchmarking.
• The diversity of questions and answers exceeds that of image captions, indicating that VQA captures a broader range of visual and linguistic understanding.
• The VQA initiative includes an annual challenge and workshop to promote community-wide progress and standardization in multimodal AI research.