[Paper Review] Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

Chenyou Fan, Xiaofan Zhang+4 more|arXiv (Cornell University)|Apr 8, 2019

Multimodal Machine Learning Applications28 references35 citations

TL;DR

The paper presents an end-to-end VideoQA framework with (i) a heterogeneous memory that fuses appearance and motion features, (ii) a redesigned question memory for complex semantic understanding, and (iii) a multimodal fusion layer enabling multi-step reasoning over visual and textual cues.

ABSTRACT

In this paper, we propose a novel end-to-end trainable Video Question Answering (VideoQA) framework with three major components: 1) a new heterogeneous memory which can effectively learn global context information from appearance and motion features; 2) a redesigned question memory which helps understand the complex semantics of question and highlights queried subjects; and 3) a new multimodal fusion layer which performs multi-step reasoning by attending to relevant visual and textual hints with self-updated attention. Our VideoQA model firstly generates the global context-aware visual and textual features respectively by interacting current inputs with memory contents. After that, it makes the attentional fusion of the multimodal visual and textual representations to infer the correct answer. Multiple cycles of reasoning can be made to iteratively refine attention weights of the multimodal data and improve the final representation of the QA pair. Experimental results demonstrate our approach achieves state-of-the-art performance on four VideoQA benchmark datasets.

Motivation & Objective

• Motivate the need to jointly model appearance and motion features for VideoQA via a heterogeneous memory architecture.
• Design a memory-augmented framework that can capture global context for both video and question representations.
• Develop a multimodal fusion module that enables multi-step reasoning over visual and textual hints.
• Demonstrate state-of-the-art performance on multiple VideoQA benchmarks and provide thorough ablations.

Proposed method

• Introduce a heterogeneous external memory that jointly writes appearance and motion content to learn co-aligned spatiotemporal attention.
• Develop a read/write mechanism with separate motion/appearance hidden states and a global memory state to produce context-aware video features.
• Incorporate a redesigned external question memory to capture complex, context-dependent semantics of questions.
• Propose a multimodal fusion layer with an LSTM controller that attends to video and question features across iterations and updates a fused representation.
• Apply multi-step reasoning (L iterations) to refine attentional weights and final QA representation.
• Train end-to-end with standard VideoQA loss functions (multi-choice and open-ended) using Adam optimizer.

Experimental results

Research questions

• RQ1 How can heterogeneous memory effectively integrate appearance and motion features to learn joint spatiotemporal attention for VideoQA?
• RQ2 Can a redesigned question memory better capture complex question semantics and highlight queried subjects?
• RQ3 Does a multimodal fusion layer with iterative reasoning improve alignment between visual content and question cues for accurate answers?
• RQ4 What is the impact of multi-step reasoning iterations on final VideoQA performance?
• RQ5 Do memory-based architectures outperform early/late fusion baselines on standard VideoQA benchmarks?

Key findings

• The proposed model achieves state-of-the-art performance on four VideoQA benchmark datasets.
• Memory modules that fuse visual features and question semantics yield improved attention and reasoning over baselines.
• Joint visual memory (appearance + motion) consistently outperforms episodic memory and simple fusion variants.
• The multimodal fusion layer with multi-step reasoning improves alignment between video content and questions, outperforming ST-VQA and Co-Mem baselines on TGIF-QA and other datasets.
• Ablation studies show that both visual memory and question memory contribute to performance gains, with further gains when used together.