[Paper Review] Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Junhua Mao, Wei Xu+4 more|arXiv (Cornell University)|Dec 20, 2014

Multimodal Machine Learning Applications39 references649 citations

TL;DR

This paper proposes a multimodal Recurrent Neural Network (m-RNN) that jointly models image and sentence representations using a deep convolutional neural network (CNN) for images and a deep recurrent neural network (RNN) for text, with a multimodal fusion layer enabling end-to-end learning. The model achieves state-of-the-art performance on image captioning and retrieval tasks across four benchmark datasets, significantly outperforming prior methods in both generation and retrieval accuracy.

ABSTRACT

In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated by sampling from this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html .

Motivation & Objective

• To develop a unified deep learning framework that generates novel, descriptive captions for images beyond retrieval-based methods.
• To improve image and sentence retrieval by jointly modeling visual and linguistic features in a shared embedding space.
• To enable end-to-end training of a multimodal system that captures both spatial visual context and sequential linguistic structure.
• To demonstrate the effectiveness of the m-RNN model on diverse benchmark datasets for captioning and retrieval.
• To explore post-processing techniques such as consensus reranking to further enhance caption quality.

Proposed method

• The m-RNN model integrates a deep CNN (VGG-16) for image feature extraction and a deep RNN with word embeddings for sentence modeling.
• A multimodal fusion layer combines visual and hidden linguistic representations using a joint projection to a shared semantic space.
• The model is trained end-to-end using a log-likelihood objective to maximize the probability of ground-truth captions given images.
• Backpropagation through time and convolutional layers enables joint optimization of all network components.
• Consensus reranking is applied post-generation using nearest neighbor captions from the training set to improve caption quality.
• For reranking, sentence similarity is computed using BLEU or CIDEr scores between generated hypotheses and reference captions from k nearest neighbors.

Experimental results

Research questions

• RQ1 Can a multimodal RNN architecture jointly model visual and linguistic sequences to generate novel, accurate image captions?
• RQ2 How does the m-RNN model compare to retrieval-based methods in terms of caption generation quality and diversity?
• RQ3 To what extent can the m-RNN model improve image and sentence retrieval performance compared to state-of-the-art embedding-based methods?
• RQ4 Can consensus reranking using nearest neighbor captions further enhance the quality of generated captions?
• RQ5 What is the impact of incorporating refined image features from the m-RNN on downstream captioning and retrieval performance?

Key findings

• The m-RNN model achieves state-of-the-art performance on image captioning, outperforming prior methods on IAPR TC-12, Flickr 8K, Flickr 30K, and MS COCO datasets.
• On the MS COCO validation set, the m-RNN model achieves a CIDEr score of 0.842, significantly higher than previous methods.
• After applying consensus reranking with nearest neighbors, the CIDEr score on the MS COCO validation set improves to 0.938, and to 0.917 on the test server.
• The model improves BLEU-4 scores by 3.5 points on the validation set and 3.3 points on the test set after consensus reranking.
• The oracle reranking performance (using ground-truth captions) reaches a CIDEr score of 1.272 on the validation set, indicating substantial room for improvement.
• Refined image features from the m-RNN model capture more accurate visual semantics, such as correctly identifying objects like bananas in images where original VGG-16 features fail.