[Paper Review] Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering

Vahid Kazemi, Ali Elqursh|arXiv (Cornell University)|Apr 11, 2017

Multimodal Machine Learning ApplicationsComputer Science2 references148 citations

TL;DR

The paper presents a simple, strong baseline for visual question answering that surpasses prior state-of-the-art on VQA 1.0 and VQA 2.0 using an LSTM question encoder, a ResNet image encoder, soft-attention over image regions, and a two-layer classifier.

ABSTRACT

This paper presents a new baseline for visual question answering task. Given an image and a question in natural language, our model produces accurate answers according to the content of the image. Our model, while being architecturally simple and relatively small in terms of trainable parameters, sets a new state of the art on both unbalanced and balanced VQA benchmark. On VQA 1.0 open ended challenge, our model achieves 64.6% accuracy on the test-standard set without using additional data, an improvement of 0.4% over state of the art, and on newly released VQA 2.0, our model scores 59.7% on validation set outperforming best previously reported results by 0.5%. The results presented in this paper are especially interesting because very similar models have been tried before but significantly lower performance were reported. In light of the new results we hope to see more meaningful research on visual question answering in the future.

Motivation & Objective

• Motivate and establish a strong, simple baseline for VQA that challenges the notion that more complex architectures are required to achieve state-of-the-art performance.
• Demonstrate that careful details in training (normalization, dropout, soft attention) yield significant gains even with a compact model.
• Quantify performance on VQA 1.0 (test-standard) and VQA 2.0 (validation) and compare against prior state-of-the-art.

Proposed method

• Encode the question with an LSTM fed by word embeddings.
• Extract image features with a pretrained 152-layer ResNet, taking the last convolutional layer outputs (14x14x2048) and applying L2 normalization.
• Apply a stacked soft-attention mechanism over spatial image features conditioned on the LSTM state to obtain multiple image glimpses.
• Concatenate the image glimpses with the final LSTM state and pass through a two-layer classifier to output probabilities over the most frequent answers (top 3000).
• Train with cross-entropy loss averaged over all correct answers per question, using Adam optimizer and dropout for regularization.

Experimental results

Research questions

• RQ1 Can a relatively simple architecture with careful training details achieve state-of-the-art results in VQA 1.0 and VQA 2.0?
• RQ2 What is the impact of normalization, dropout, attention, and architectural choices on VQA performance?
• RQ3 How does the proposed baseline compare to existing methods on standard VQA benchmarks?
• RQ4 Is soft-attention essential for performance gains in VQA models?
• RQ5 What are the effects of hyperparameters (embedding size, LSTM size, attention size, classifier size) on accuracy?

Key findings

• Achieves 64.6% test-standard accuracy on VQA 1.0 without extra data, outperforming prior best by 0.4%.
• Scores 59.7% on VQA 2.0 validation, outperforming prior best by 0.5%.
• L2 normalization of image features, dropout, and soft-attention significantly improve accuracy and training efficiency.
• Stacked attention offers limited gains compared to a strong single baseline; a two-layer classifier considerably helps performance.
• The model uses ResNet-based image embeddings and a 1024-dim LSTM with 300-dim word embeddings; several hyperparameters show limited impact within reasonable ranges.