[Paper Review] Hierarchical Question-Image Co-Attention for Visual Question Answering

Jiasen Lu, Jianwei Yang+2 more|arXiv (Cornell University)|May 31, 2016

Multimodal Machine Learning Applications24 references1,216 citations

TL;DR

This paper introduces a hierarchical co-attention model that jointly attends to image regions and question words at word, phrase, and question levels, using parallel or alternating co-attention, to improve Visual Question Answering performance.

ABSTRACT

A number of recent works have proposed attention models for Visual Question Answering (VQA) that generate spatial maps highlighting image regions relevant to answering the question. In this paper, we argue that in addition to modeling "where to look" or visual attention, it is equally important to model "what words to listen to" or question attention. We present a novel co-attention model for VQA that jointly reasons about image and question attention. In addition, our model reasons about the question (and consequently the image via the co-attention mechanism) in a hierarchical fashion via a novel 1-dimensional convolution neural networks (CNN). Our model improves the state-of-the-art on the VQA dataset from 60.3% to 60.5%, and from 61.6% to 63.3% on the COCO-QA dataset. By using ResNet, the performance is further improved to 62.1% for VQA and 65.4% for COCO-QA.

Motivation & Objective

• Motivate and address both visual attention (where to look) and question attention (which words to listen to) in VQA.
• Propose a co-attention mechanism that jointly reasons about image and question attention.
• Introduce a three-level hierarchical question representation (word, phrase, question) with a novel phrase-level convolution-pooling strategy.
• Develop parallel and alternating co-attention variants and validate on large VQA and COCO-QA datasets.

Proposed method

• Define a multi-modal affinity C between image regions V and question words Q.
• Compute co-attended features at word, phrase, and question levels using parallel co-attention (equations 3-5) or alternating co-attention (section 3.3 and equations 6).
• Construct phrase representations via 1-D CNNs across unigram, bigram, and trigram windows with max-pooling (equations 1-2).
• Encode the hierarchical question representations with word embeddings, phrase-level LSTM, and question-level LSTM (Sec. 3.2).
• Recursively fuse attended image and question features from all levels through an MLP to predict the final answer (Eq. 7 and Sec. 3.4).
• Evaluate on VQA and COCO-QA datasets and compare to state-of-the-art methods; analyze ablations (Tables 1–3) and provide qualitative visualizations (Fig. 4–6).

Experimental results

Research questions

• RQ1 Can joint image-question co-attention improve VQA performance beyond visual attention alone?
• RQ2 Does incorporating a three-level hierarchical question representation (word, phrase, question) improve the quality of attended features and final answers?
• RQ3 How do parallel and alternating co-attention strategies compare in terms of performance and training stability?
• RQ4 What is the impact of phrase-level convolution-pooling versus non-convolution phrase representations?

Key findings

• The proposed co-attention model improves state-of-the-art results on VQA and COCO-QA datasets when using ResNet features.
• Parallel co-attention generally performs better than alternating co-attention on COCO-QA, and both outperform baselines with image-only attention.
• A three-level (word, phrase, question) hierarchy with co-attention at each level yields gains over models without linguistic attention, with the top (question) level contributing most to performance.
• Phrase-level convolution-pooling adaptively selects informative n-grams, enhancing phrase representations compared to fixed-sized windows.
• Ablation studies show that removing any level of attention degrades performance, with question-level attention having the largest impact, indicating its proximity to the answer predictor.
• Qualitative visualizations demonstrate interpretable co-attention maps aligning with relevant image regions and question phrases.