[Paper Review] Ask Your Neurons: A Neural-based Approach to Answering Questions about Images

Mateusz Malinowski, Marcus Rohrbach+1 more|arXiv (Cornell University)|May 5, 2015

Multimodal Machine Learning Applications1 references186 citations

TL;DR

This paper proposes Neural-Image-QA, a joint end-to-end neural network combining CNNs for image features and LSTMs for question encoding and answer generation, achieving state-of-the-art performance by doubling the accuracy of prior methods on visual question answering. It introduces DAQUAR-Consensus with multiple human answers, new consensus-based metrics, and demonstrates that a language-only variant outperforms a human baseline, indicating learned common sense in the model.

ABSTRACT

We address a question answering task on real-world images that is set up as a Visual Turing Test. By combining latest advances in image representation and natural language processing, we propose Neural-Image-QA, an end-to-end formulation to this problem for which all parts are trained jointly. In contrast to previous efforts, we are facing a multi-modal problem where the language output (answer) is conditioned on visual and natural language input (image and question). Our approach Neural-Image-QA doubles the performance of the previous best approach on this problem. We provide additional insights into the problem by analyzing how much information is contained only in the language part for which we provide a new human baseline. To study human consensus, which is related to the ambiguities inherent in this challenging task, we propose two novel metrics and collect additional answers which extends the original DAQUAR dataset to DAQUAR-Consensus.

Motivation & Objective

• To develop an end-to-end neural network for visual question answering that jointly learns from images and natural language questions.
• To improve performance on the DAQUAR benchmark beyond prior state-of-the-art methods.
• To study human consensus and ambiguity in visual question answering by collecting additional reference answers.
• To introduce new evaluation metrics sensitive to human disagreement: Average Consensus and Min Consensus.
• To investigate whether language-only models can outperform human baselines, indicating learned common sense.

Proposed method

• A CNN (e.g., GoogLeNet) extracts deep visual features from input images, which are then fused with question embeddings.
• A Long Short-Term Memory (LSTM) network encodes the natural language question and generates answers autoregressively, conditioned on both image and question features.
• The entire model is trained end-to-end using backpropagation, with word-level cross-entropy loss for answer generation.
• The system is evaluated using standard metrics (accuracy, WUPS) and two novel consensus-based metrics: Average Consensus and Min Consensus.
• A language-only variant is trained without visual input to assess the model’s ability to infer answers from language alone.
• The DAQUAR dataset is extended to DAQUAR-Consensus by collecting additional human answers to study inter-annotator agreement and ambiguity.

Experimental results

Research questions

• RQ1 Can a joint end-to-end neural network architecture outperform prior methods in visual question answering?
• RQ2 To what extent can a language-only model generate plausible answers without visual input, and how does it compare to human performance?
• RQ3 How much ambiguity exists in human answers to visual questions, and how can this be quantitatively measured?
• RQ4 Do consensus-based evaluation metrics better reflect human disagreement than standard metrics?
• RQ5 What are the failure modes of current visual question answering models, particularly for spatial reasoning and rare object categories?

Key findings

• Neural-Image-QA achieves a 60.50% accuracy on the DAQUAR-Consensus dataset using the Min Consensus metric, significantly outperforming prior work.
• The model improves performance by doubling the accuracy of the previous best approach on the full DAQUAR dataset.
• A language-only variant of the model achieves 36.78% accuracy on the original DAQUAR test set, outperforming a new human baseline collected under the same conditions.
• The Min Consensus metric applied to human answers improves the human baseline score from 20% to 60.50% on the same test set, indicating high inter-annotator disagreement.
• The model struggles with spatial reasoning (21 WUPS@0.9), small objects, negations, and shape recognition, with scores below 12 WUPS@0.9.
• Failure cases include strong occlusion, out-of-vocabulary answers (e.g., 'toaster'), and unusual instances, highlighting limitations of global CNN features.