[Paper Review] Textually Enriched Neural Module Networks for Visual Question Answering

Khyathi Raghavi Chandu, Mary Arpita Pyreddy+2 more|arXiv (Cornell University)|Sep 23, 2018

Multimodal Machine Learning Applications26 references7 citations

TL;DR

This paper proposes a textually enriched neural module network (NMN) for visual question answering by integrating image captions and external knowledge to mitigate information loss during question parsing. By attending to relevant caption content and leveraging external knowledge bases, the model improves accuracy to 57.1% on VQA 1.0 test-dev, with notable gains in questions requiring contextual or commonsense reasoning.

ABSTRACT

Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.

Motivation & Objective

• Address information loss in neural module networks (NMNs) when simplifying questions into parse trees.
• Improve VQA performance by incorporating image captions as supplementary textual context to enhance visual-linguistic reasoning.
• Reduce noise from irrelevant caption content by introducing attention mechanisms to focus on salient caption segments.
• Enhance answers to knowledge-intensive questions by integrating external knowledge bases beyond image and caption data.
• Evaluate the impact of caption-based attention and knowledge integration on open-ended VQA benchmarks, particularly for complex or ambiguous queries.

Proposed method

• Integrate image captions into the NMN framework using an LSTM to encode caption text and attend to relevant parts via soft attention over caption tokens.
• Apply a soft attention mechanism that aligns question words with relevant caption phrases, filtering out irrelevant caption content to reduce noise.
• Extend the NMN architecture with a Measure module for counting questions, replacing the Describe module to improve numerical reasoning.
• Leverage external knowledge bases to enrich visual features for questions requiring commonsense or factual knowledge beyond the image.
• Use a late-fusion mechanism where image features (from CNN), question features (from LSTM), and caption features (from LSTM with attention) are combined via element-wise multiplication and followed by a softmax layer for answer prediction.
• Train the entire model end-to-end using backpropagation, with attention weights and knowledge retrieval parameters jointly optimized.

Experimental results

Research questions

• RQ1 Can integrating image captions improve the performance of neural module networks in visual question answering by preserving linguistic information lost during parse simplification?
• RQ2 Does attending selectively to relevant parts of a caption rather than using the full caption reduce noise and improve answer accuracy?
• RQ3 To what extent does incorporating external knowledge bases enhance reasoning for questions requiring factual or commonsense knowledge?
• RQ4 How does the proposed caption-attention mechanism compare to standard NMNs in handling ambiguous or complex questions?
• RQ5 What types of errors persist in the model, and how are they influenced by generic terms in captions that may lead to spurious associations?

Key findings

• The proposed caption-attention mechanism improved test-dev accuracy by 1.2 percentage points (from 55.9% to 57.1%) on the VQA 1.0 benchmark.
• The model achieved a 1.9% improvement on the training set, indicating better generalization through enriched textual supervision.
• In cases where the caption was relevant to the question, such as 'two planes flying' or 'a pizza', the model correctly predicted answers like '2' and 'pizza', outperforming standard NMNs.
• The model occasionally failed when captions contained generic or misleading terms—e.g., associating 'group' with a high count (5 instead of 6) due to training data biases.
• Attention over captions did not improve performance when the caption was irrelevant to the question, highlighting the risk of noise from poor caption quality.
• The Measure module did not improve performance over the Describe module, likely due to information loss in attention maps or insufficient image feature utilization.