[논문 리뷰] Dynamic Memory Networks for Visual and Textual Question Answering

Caiming Xiong, Stephen Merity외 1명|arXiv (Cornell University)|2016. 03. 04.

Multimodal Machine Learning Applications참고 문헌 3인용 수 593

한 줄 요약

본 논문은 Dynamic Memory Networks (DMN)을 시각적 질문 응답을 다루도록 확장하고 이미지 입력 모듈과 메모리 및 입력 표현을 개선하여 VQA와 bAbI-10k에서 지원 사실의 감독 없이도 최첨단 성능을 달성한다.

ABSTRACT

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the \babi-10k text question-answering dataset without supporting fact supervision.

연구 동기 및 목표

• Extend DMN to handle visual and textual question answering without requiring labeled supporting facts.
• Improve the input representation to enable better interaction and global context for both text and images.
• Enhance the memory update mechanism to better support multi-pass episodic reasoning.
• Demonstrate state-of-the-art performance on both the VQA dataset and the bAbI-10k text QA dataset.

제안 방법

• Introduce an input fusion layer in the text module to allow interactions between sentences via a bi-directional GRU.
• Develop an input module for images that splits images into 14x14 local regions, projects them into the textual feature space, and applies a bi-directional GRU over regions for global context.
• Replace the standard DMN attention with an attention-based GRU that uses the attention gates to update hidden states (Eq. 11).
• Update the episodic memory by feeding the contextual vector c^t and previous memory through a memory update (Eq. 12) and optionally a ReLU-based untied update (Eq. 13).
• Experiment with both soft attention and the attention-based GRU, selecting the latter for DMN+.
• Train and evaluate on bAbI-10k, DAQUAR-ALL, and VQA datasets to compare against state-of-the-art approaches.

실험 결과

연구 질문

• RQ1 Can DMN be extended to visual question answering without annotated supporting facts?
• RQ2 Do improvements to the input module and memory updates generalize across text QA and VQA tasks?
• RQ3 How do different attention mechanisms (soft vs. attention-based GRU) affect reasoning in DMN+?
• RQ4 Does untied memory weighting help or hinder performance across tasks?

주요 결과

• DMN+ achieves higher accuracy on DAQUAR-ALL and VQA compared to prior DMN variants without requiring labeled supporting facts.
• The input fusion layer improves interaction between distant facts/sentences and between image regions, boosting both textual and visual QA performance.
• The attention-based GRU improves handling of questions requiring complex positional or ordering reasoning, particularly in text QA.
• Untied memory weights with a ReLU memory update provide additional gains on average but can cause overfitting on some tasks.
• Overall, DMN+ delivers state-of-the-art results on both the VQA and bAbI-10k datasets, surpassing end-to-end memory networks and neural reasoners on several tasks.