[Paper Review] Dynamic Memory Networks for Visual and Textual Question Answering

Caiming Xiong, Stephen Merity+1 more|arXiv (Cornell University)|Mar 4, 2016

Multimodal Machine Learning Applications3 references593 citations

TL;DR

The paper extends Dynamic Memory Networks (DMN) to handle visual question answering with an image input module and improves memory and input representations, achieving state-of-the-art results on VQA and bAbI-10k without supervising supporting facts.

ABSTRACT

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.

Motivation & Objective

• Extend DMN to handle visual and textual question answering without requiring labeled supporting facts.
• Improve the input representation to enable better interaction and global context for both text and images.
• Enhance the memory update mechanism to better support multi-pass episodic reasoning.
• Demonstrate state-of-the-art performance on both the VQA dataset and the bAbI-10k text QA dataset.

Proposed method

• Introduce an input fusion layer in the text module to allow interactions between sentences via a bi-directional GRU.
• Develop an input module for images that splits images into 14x14 local regions, projects them into the textual feature space, and applies a bi-directional GRU over regions for global context.
• Replace the standard DMN attention with an attention-based GRU that uses the attention gates to update hidden states (Eq. 11).
• Update the episodic memory by feeding the contextual vector c^t and previous memory through a memory update (Eq. 12) and optionally a ReLU-based untied update (Eq. 13).
• Experiment with both soft attention and the attention-based GRU, selecting the latter for DMN+.
• Train and evaluate on bAbI-10k, DAQUAR-ALL, and VQA datasets to compare against state-of-the-art approaches.

Experimental results

Research questions

• RQ1 Can DMN be extended to visual question answering without annotated supporting facts?
• RQ2 Do improvements to the input module and memory updates generalize across text QA and VQA tasks?
• RQ3 How do different attention mechanisms (soft vs. attention-based GRU) affect reasoning in DMN+?
• RQ4 Does untied memory weighting help or hinder performance across tasks?

Key findings

• DMN+ achieves higher accuracy on DAQUAR-ALL and VQA compared to prior DMN variants without requiring labeled supporting facts.
• The input fusion layer improves interaction between distant facts/sentences and between image regions, boosting both textual and visual QA performance.
• The attention-based GRU improves handling of questions requiring complex positional or ordering reasoning, particularly in text QA.
• Untied memory weights with a ReLU memory update provide additional gains on average but can cause overfitting on some tasks.
• Overall, DMN+ delivers state-of-the-art results on both the VQA and bAbI-10k datasets, surpassing end-to-end memory networks and neural reasoners on several tasks.