[Paper Review] Explain Images with Multimodal Recurrent Neural Networks

Junhua Mao, Wei Xu+3 more|arXiv (Cornell University)|Oct 4, 2014

Generative Adversarial Networks and Image Synthesis32 references369 citations

TL;DR

This paper proposes a multimodal Recurrent Neural Network (m-RNN) that jointly models image and sentence generation by fusing deep convolutional features from images with recurrent language modeling, achieving state-of-the-art performance in image captioning and cross-modal retrieval across IAPR TC-12, Flickr 8K, and Flickr 30K datasets.

ABSTRACT

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel sentence descriptions to explain the content of images. It directly models the probability distribution of generating a word given previous words and the image. Image descriptions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on three benchmark datasets (IAPR TC-12, Flickr 8K, and Flickr 30K). Our model outperforms the state-of-the-art generative method. In addition, the m-RNN model can be applied to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval.

Motivation & Objective

• To develop a unified deep learning framework that generates natural language descriptions for images and supports image-sentence retrieval.
• To overcome limitations of retrieval-based methods that cannot describe novel image compositions not present in training data.
• To integrate visual features from a deep CNN with sequential language modeling via a multimodal recurrent architecture.
• To jointly optimize for both image captioning and cross-modal retrieval using a perplexity-based objective function.
• To demonstrate the model's generalization and extensibility by applying it to diverse benchmark datasets with consistent performance gains.

Proposed method

• The m-RNN model consists of three components: a deep convolutional neural network (CNN) for image feature extraction, a recurrent neural network (RNN) for language modeling, and a multimodal fusion layer that combines visual and linguistic representations.
• The multimodal layer enables interaction between image features and hidden states of the RNN, allowing the model to condition word generation on both prior words and the full image content.
• The model is trained end-to-end using a perplexity-based loss function that maximizes the likelihood of ground-truth captions given the image.
• Backpropagation through time and across modalities updates all network parameters simultaneously, including CNN, RNN, and multimodal fusion weights.
• The architecture supports arbitrary-length context via the RNN's hidden state, unlike fixed-context models such as Log-Bilinear.
• The model is evaluated on both image captioning (via BLEU and perplexity) and cross-modal retrieval (via R@K and median rank).

Experimental results

Research questions

• RQ1 Can a unified multimodal RNN architecture jointly improve image captioning and cross-modal retrieval performance compared to separate models?
• RQ2 Does incorporating recurrent modeling of language context improve caption generation over fixed-context or non-recurrent models?
• RQ3 Can the m-RNN model generate novel, descriptive captions for unseen image compositions not present in training data?
• RQ4 How does the m-RNN perform on standard benchmarks like IAPR TC-12, Flickr 8K, and Flickr 30K in both captioning and retrieval tasks?
• RQ5 To what extent does the model benefit from using more advanced image features (e.g., object detection features) in future extensions?

Key findings

• On the IAPR TC-12 dataset, the m-RNN achieved 20.9% R@1 and 13.2% R@1 for image-to-text and text-to-image retrieval, respectively, significantly outperforming baseline methods.
• On Flickr 8K, the m-RNN achieved 14.5% R@1 and 11.5% R@1 for image-to-text and text-to-image retrieval, surpassing state-of-the-art methods like DeepFE-decaf and Socher-decaf.
• On Flickr 30K, the m-RNN achieved 18.4% R@1 and 12.6% R@1 for image-to-text and text-to-image retrieval, outperforming DeViSE-avg-rcnn and DeepFE-rcnn.
• In image captioning, the m-RNN achieved a perplexity of 24.39 on Flickr 8K and 35.11 on Flickr 30K, significantly lower than the RNN baseline (30.39 and 43.96), indicating better language modeling.
• The m-RNN also achieved higher BLEU scores: 0.5778 (B-1), 0.2751 (B-2), and 0.2307 (B-3) on Flickr 8K, compared to the baseline RNN's 0.4383, 0.1849, and 0.1339.
• The model demonstrated robustness and generalization, generating coherent and contextually relevant captions even on unseen image compositions, as shown in qualitative examples.