[Paper Review] Learning a Recurrent Visual Representation for Image Caption Generation

Xinlei Chen, C. Lawrence Zitnick|arXiv (Cornell University)|Nov 20, 2014

Multimodal Machine Learning Applications35 references180 citations

TL;DR

This paper proposes a bi-directional recurrent neural network model that learns a shared visual-semantic representation for image captioning, enabling both novel caption generation from images and visual feature reconstruction from text. By introducing a recurrent visual memory that dynamically encodes seen concepts, the model achieves state-of-the-art performance on image captioning, with human preference for generated captions surpassing human-written ones in 19.8% of comparisons on MS COCO.

ABSTRACT

In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both sentences and images to a common embedding, we enable the generation of novel sentences given an image. Using the same model, we can also reconstruct the visual features associated with an image given its visual description. We use a novel recurrent visual memory that automatically learns to remember long-term visual concepts to aid in both sentence generation and visual feature reconstruction. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are preferred by humans over $19.8\%$ of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.

Motivation & Objective

• To develop a bi-directional model that can generate novel image captions from visual input and reconstruct visual features from textual descriptions.
• To address the limitation of prior models that cannot perform inverse mapping or generate novel sentences from images.
• To improve long-term memory of visual concepts during sentence generation using a recurrent visual memory mechanism.
• To achieve state-of-the-art performance on image captioning, image retrieval, and sentence retrieval tasks.
• To evaluate human preference of generated captions against human-written captions in a direct comparison.

Proposed method

• The model uses a recurrent neural network (RNN) to encode image features into a dynamic visual memory that evolves as words are generated or processed.
• A novel recurrent visual memory component learns to reconstruct the original visual features from the sequence of words, enabling bi-directional mapping.
• The model is trained end-to-end to maximize the likelihood of ground-truth captions while minimizing reconstruction error between original and reconstructed visual features.
• For image-to-text generation, the visual memory is initialized with image features and updated at each decoding step using word embeddings.
• For text-to-image reconstruction, the visual memory is initialized with a zero vector and updated using the RNN's hidden state as words are processed.
• The model uses both likelihood of the caption given the image (T) and reconstruction error of visual features (I) for ranking in retrieval tasks.

Experimental results

Research questions

• RQ1 Can a single deep learning model generate novel, coherent image captions from visual input while also reconstructing visual features from text?
• RQ2 Does a recurrent visual memory mechanism improve long-term retention of visual concepts during sequence generation?
• RQ3 How does the performance of the proposed model compare to human-annotated captions in human preference evaluations?
• RQ4 Can the model achieve state-of-the-art results on both image and sentence retrieval tasks using the same visual features?
• RQ5 Does combining caption likelihood and visual reconstruction error improve retrieval performance over using either metric alone?

Key findings

• The model achieves state-of-the-art results on image caption generation, with human preference for its generated captions exceeding human-written captions in 19.8% of comparisons on the MS COCO dataset.
• On the PASCAL 1K dataset, the model achieves SOTA performance on both BLEU and METEOR metrics for image captioning.
• For image and sentence retrieval, the model achieves results comparable or better than previous state-of-the-art methods, especially when using the same visual features (e.g., DECAF features).
• The combination of caption likelihood and visual reconstruction error (T+I) outperforms using either metric alone in retrieval tasks, improving recall and reducing median/mean rank.
• The model demonstrates strong bi-directional capability: it can generate novel captions from images and reconstruct visual features from descriptions using the same architecture and parameters.
• The recurrent visual memory mechanism enables the model to maintain and update long-term visual representations, improving coherence and accuracy in caption generation.