[Paper Review] Task-driven Visual Saliency and Attention-based Visual Question Answering

Yuetan Lin, Zhangyang Pang+2 more|arXiv (Cornell University)|Feb 22, 2017

Multimodal Machine Learning Applications32 references22 citations

TL;DR

This paper proposes a task-driven visual saliency mechanism and a novel element-wise multiplication-based attention method for visual question answering (VQA). By leveraging bidirectional LSTM to compute region-wise saliency weights based on interrelated image regions, and using element-wise multiplication to fuse visual and textual features, the model achieves state-of-the-art performance on COCO-VQA, with 63.69% accuracy on multiple-choice tasks and 56.02% on open-ended tasks, outperforming several attention-based baselines.

ABSTRACT

Visual question answering (VQA) has witnessed great progress since May, 2015 as a classic problem unifying visual and textual data into a system. Many enlightening VQA works explore deep into the image and question encodings and fusing methods, of which attention is the most effective and infusive mechanism. Current attention based methods focus on adequate fusion of visual and textual features, but lack the attention to where people focus to ask questions about the image. Traditional attention based methods attach a single value to the feature at each spatial location, which losses many useful information. To remedy these problems, we propose a general method to perform saliency-like pre-selection on overlapped region features by the interrelation of bidirectional LSTM (BiLSTM), and use a novel element-wise multiplication based attention method to capture more competent correlation information between visual and textual features. We conduct experiments on the large-scale COCO-VQA dataset and analyze the effectiveness of our model demonstrated by strong empirical results.

Motivation & Objective

• To address the lack of attention to where humans focus when asking questions about images in existing VQA models.
• To improve VQA performance by pre-selecting salient image regions based on task-specific interest patterns.
• To enhance feature fusion between vision and language modalities using a novel element-wise multiplication attention mechanism.
• To demonstrate that region pre-selection and refined attention mechanisms can significantly boost VQA accuracy without complex architectures.

Proposed method

• Proposes a task-driven visual saliency mechanism that computes region importance using bidirectional LSTM (BiLSTM) to model interrelations among region features.
• Applies BiLSTM to encode spatial relationships across image regions, generating saliency weights that highlight likely regions of interest for question-asking.
• Introduces an element-wise multiplication-based attention mechanism that fuses visual and textual features more effectively than concatenation or standard attention.
• Uses max pooling on the element-wise product of visual and textual features to generate a compact, richly representative fused vector for answer prediction.
• Employs a simple yet effective VQA baseline model enhanced with the proposed pre-selection and attention mechanisms, without requiring complex multi-step attention layers.
• Uses 3×3 region grids for computational efficiency, though acknowledges limitations in capturing small objects.

Experimental results

Research questions

• RQ1 How can VQA models better simulate human visual attention by identifying regions likely to be the focus of questions?
• RQ2 Can a BiLSTM-based mechanism effectively model interdependencies among image regions to generate task-specific saliency maps?
• RQ3 Does element-wise multiplication of visual and textual features yield better fusion than concatenation or standard attention in VQA?
• RQ4 To what extent does pre-selecting salient regions improve VQA performance, especially on open-ended and multiple-choice questions?

Key findings

• The proposed model achieves 63.69% accuracy on the multiple-choice task and 56.02% on the open-ended task of the COCO-VQA test-dev set, outperforming several state-of-the-art models.
• On the test-standard set, the model reaches 63.99% accuracy on multiple-choice and 56.42% on open-ended tasks, demonstrating strong generalization.
• The model performs particularly well on the 'other' answer type, which includes object and scene recognition, indicating effective region-of-interest detection.
• Visualization confirms that the pre-selection mechanism highlights semantically relevant regions (e.g., people, objects of interest), and attention maps align with question semantics (e.g., focusing on clothing for weather-related questions).
• The method is generalizable and can be integrated into other attention-based VQA models to improve their performance.
• Despite using only 3×3 region features (fewer than 100 or 14×14 in other models), the model achieves competitive results, suggesting efficiency and effectiveness.