[Paper Review] Simple Baseline for Visual Question Answering

Bolei Zhou, Yuandong Tian+3 more|arXiv (Cornell University)|Dec 7, 2015

Multimodal Machine Learning ApplicationsComputer Science19 references292 citations

TL;DR

This paper proposes a simple bag-of-words baseline model (iBOWIMG) that concatenates question word embeddings and pre-extracted CNN features from images to predict answers via a softmax classifier. Despite its simplicity, the model achieves performance comparable to complex recurrent neural network-based approaches on the COCO VQA dataset, demonstrating that strong performance can be achieved without attention mechanisms or RNNs, and revealing insights into model interpretability through attention visualization and word contribution analysis.

ABSTRACT

We describe a very simple bag-of-words baseline for visual question answering. This baseline concatenates the word features from the question and CNN features from the image to predict the answer. When evaluated on the challenging VQA dataset [2], it shows comparable performance to many recent approaches using recurrent neural networks. To explore the strength and weakness of the trained model, we also provide an interactive web demo and open-source code. .

Motivation & Objective

• To evaluate the performance of a minimal, non-recurrent baseline model for visual question answering.
• To investigate whether simple feature concatenation of question words and image features can match complex models on the VQA benchmark.
• To analyze the model's learned correlations between question words, image content, and predicted answers.
• To provide interpretability through word importance ranking and class activation mapping (CAM) for visual attention.
• To enable interactive exploration of VQA models via a publicly available web demo and code.

Proposed method

• The model uses a one-hot encoding of question words followed by a learned word embedding layer to produce text features.
• Image features are extracted from a pre-trained GoogLeNet network as deep CNN features.
• The word and image features are concatenated and fed into a single softmax layer for multi-class classification of answer labels.
• The model is trained using cross-entropy loss with stochastic gradient descent on the COCO VQA dataset.
• Word importance is computed by backpropagating the softmax output to the word embedding layer, revealing which words most influence the prediction.
• Class activation mapping (CAM) is applied to visualize the most relevant image regions for each predicted answer, leveraging the linear relationship between the final layer and convolutional features.

Experimental results

Research questions

• RQ1 Can a simple bag-of-words model with no recurrent or attention mechanisms achieve competitive performance on the VQA benchmark?
• RQ2 What is the relative contribution of question words versus visual features to the model’s predictions?
• RQ3 How can we interpret the model’s predictions using word-level importance and spatial attention maps?
• RQ4 Can a minimal model reveal biases in the dataset, such as frequent word-answer associations?
• RQ5 Does the model’s implicit attention, derived from CAM, resemble that of more complex models with explicit attention mechanisms?

Key findings

• The iBOWIMG baseline achieves performance comparable to state-of-the-art recurrent neural network-based models on the COCO VQA dataset, despite its simplicity.
• The model shows strong reliance on question words: for example, 'what is the color of the sofa' is correctly answered based on the word 'color' alone, without image input.
• Word importance analysis reveals that specific words like 'doing' or 'eating' are highly predictive of answers such as 'texting' or 'hot dog'.
• Class activation mapping (CAM) generates spatial attention maps that highlight relevant image regions, such as a cellphone for 'texting' or a hot dog for 'hot dog', demonstrating implicit spatial selectivity.
• The model's predictions are often driven by frequent word-answer correlations in the dataset, indicating a potential bias toward linguistic patterns rather than visual reasoning.
• The web demo and open-source code enable interactive exploration of model behavior, revealing strengths (e.g., fast, accurate for simple questions) and weaknesses (e.g., failure on reasoning or ambiguous questions).