[Paper Review] Multimodal Memory Modelling for Video Captioning

Junbo Wang, Wei Wang+3 more|arXiv (Cornell University)|Nov 17, 2016

Multimodal Machine Learning Applications42 references8 citations

TL;DR

This paper proposes a Multimodal Memory Model (M³) for video captioning that uses a shared memory to model long-term visual-textual dependencies and guide global visual attention. By integrating external memory with LSTM-based decoding and CNN-based video encoding, M³ improves performance on MSVD and MSR-VTT, outperforming state-of-the-art methods by up to 26.9% in BLEU@4 and 8.5% in METEOR.

ABSTRACT

Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), video captioning has made great progress. However, learning an effective mapping from visual sequence space to language space is still a challenging problem. In this paper, we propose a Multimodal Memory Model (M3) to describe videos, which builds a visual and textual shared memory to model the long-term visual-textual dependency and further guide global visual attention on described targets. Specifically, the proposed M3 attaches an external memory to store and retrieve both visual and textual contents by interacting with video and sentence with multiple read and write operations. First, text representation in the Long Short-Term Memory (LSTM) based text decoder is written into the memory, and the memory contents will be read out to guide an attention to select related visual targets. Then, the selected visual information is written into the memory, which will be further read out to the text decoder. To evaluate the proposed model, we perform experiments on two publicly benchmark datasets: MSVD and MSR-VTT. The experimental results demonstrate that our method outperforms the state-of-theart methods in terms of BLEU and METEOR.

Motivation & Objective

• Address the challenge of modeling long-term visual-textual dependencies in video captioning, which existing RNN-based models struggle with due to vanishing gradients in long sequences.
• Improve global visual attention by explicitly guiding the selection of relevant visual targets through a shared memory mechanism.
• Overcome limitations of single visual representations and attention mechanisms that rely solely on LSTM hidden states.
• Demonstrate the effectiveness of multimodal memory in enhancing video captioning performance across diverse benchmark datasets.
• Investigate the impact of different visual feature extractors on captioning quality and memory integration.

Proposed method

• Introduce a multimodal memory matrix (Mem) that stores and retrieves both visual and textual representations through multiple read and write operations.
• Use a CNN-based video encoder (e.g., Inception-V3, ResNet, VGG-19) to extract frame-level features, forming the input video representation.
• Implement an LSTM-based text decoder that generates words conditioned on both previous hidden states and content read from the multimodal memory.
• Guide visual attention using memory-read content instead of LSTM hidden states, enabling more accurate and global target selection.
• Perform two write operations per decoding step: (1) write LSTM decoder hidden states to memory, and (2) write selected visual features (via attention) to memory.
• Enable bidirectional memory interaction: read from memory to guide attention and decoder, and write to memory to preserve and update multimodal context.

Experimental results

Research questions

• RQ1 Can a shared multimodal memory effectively model long-term dependencies between visual and textual sequences in video captioning?
• RQ2 Does using memory-read content to guide visual attention improve the relevance and accuracy of generated captions compared to attention based on LSTM hidden states?
• RQ3 How does multimodal memory integration affect performance across different visual feature extractors (e.g., Inception-V3, VGG-19, C3D)?
• RQ4 Can the proposed M³ model consistently outperform state-of-the-art methods on diverse benchmarks like MSVD and MSR-VTT?
• RQ5 What is the contribution of motion-aware features (e.g., C3D) versus appearance-only features (e.g., VGG-19) when combined with multimodal memory?

Key findings

• The M³ model achieves a 26.9% relative improvement in BLEU@4 and 8.5% in METEOR over the baseline SA model when using GoogleNet features on MSVD.
• M³ with Inception-V3 features (M³-inception) achieves the best performance on MSVD, outperforming models using VGG-19, ResNet, and GoogleNet features.
• On MSR-VTT, M³ models consistently outperform their corresponding SA baselines, with M³-VC (using VGG-19 and C3D features) achieving higher BLEU and METEOR scores than all compared methods.
• The use of C3D features (which encode motion) leads to better performance than VGG-19 features on MSR-VTT, indicating motion information is critical for this dataset.
• Visual inspection confirms that M³ generates more precise object terms (e.g., "basketball" instead of "soccer ball"), and applies global attention to relevant targets (e.g., "dog" instead of "guitar").
• The performance ranking of models using different visual features closely correlates with their ImageNet classification accuracy, confirming the importance of high-quality visual representations.