[Paper Review] Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Ryan Kiros, Ruslan Salakhutdinov+1 more|arXiv (Cornell University)|Nov 10, 2014

Multimodal Machine Learning Applications47 references1,320 citations

TL;DR

This paper proposes a unified encoder-decoder framework that learns a joint visual-semantic embedding space using LSTM-based sentence encoding and a novel structure-content neural language model (SC-NLM) for decoding. It achieves state-of-the-art performance on Flickr8K and Flickr30K without object detection, and sets new SOTA results with deeper features, while demonstrating linguistic regularities in multimodal vector spaces via vector arithmetic.

ABSTRACT

Inspired by recent advances in multimodal learning and machine translation, we introduce an encoder-decoder pipeline that learns (a): a multimodal joint embedding space with images and text and (b): a novel language model for decoding distributed representations from our space. Our pipeline effectively unifies joint image-text embedding models with multimodal neural language models. We introduce the structure-content neural language model that disentangles the structure of a sentence to its content, conditioned on representations produced by the encoder. The encoder allows one to rank images and sentences while the decoder can generate novel descriptions from scratch. Using LSTM to encode sentences, we match the state-of-the-art performance on Flickr8K and Flickr30K without using object detections. We also set new best results when using the 19-layer Oxford convolutional network. Furthermore we show that with linear encoders, the learned embedding space captures multimodal regularities in terms of vector space arithmetic e.g. *image of a blue car* - "blue" + "red" is near images of red cars. Sample captions generated for 800 images are made available for comparison.

Motivation & Objective

• To unify joint image-text embedding models with multimodal neural language models for end-to-end image caption generation.
• To develop a structure-content neural language model (SC-NLM) that disentangles sentence structure from content, conditioned on multimodal embeddings.
• To enable both image-sentence ranking and novel caption generation using a single unified framework.
• To investigate whether multimodal vector spaces support linguistic regularities analogous to those in monolingual word embeddings.
• To improve caption quality beyond template-based and composition-based methods through neural sequence generation.

Proposed method

• Uses an LSTM encoder to embed sentences into a shared visual-semantic space, with image features projected into the same space via a linear projection.
• Employs a pairwise ranking loss to optimize the joint embedding space, ensuring semantically similar image-sentence pairs are close in vector space.
• Introduces the structure-content neural language model (SC-NLM), which conditions generation on encoder outputs while disentangling syntactic structure from semantic content.
• Applies linear sentence encoding (sum of word vectors) to enable vector arithmetic in the multimodal space, such as *image of a blue car* − "blue" + "red" ≈ images of red cars.
• Trains the encoder-decoder pipeline on a combination of Flickr30K and MS-COCO datasets, using features from a 19-layer Oxford convolutional network for improved performance.
• Uses PCA projections and nearest-neighbor retrieval to qualitatively analyze analogies and structural regularities in the learned multimodal space.

Experimental results

Research questions

• RQ1 Can a unified encoder-decoder framework effectively combine image-text embedding learning with neural caption generation?
• RQ2 Does the proposed structure-content neural language model (SC-NLM) produce more fluent and diverse captions than prior methods?
• RQ3 Can multimodal vector spaces support linguistic regularities such as analogical reasoning, demonstrated via vector arithmetic?
• RQ4 How does the performance of the model compare to state-of-the-art methods on standard image captioning benchmarks like Flickr8K and Flickr30K?
• RQ5 To what extent can linear encoders preserve linguistic regularities in multimodal embeddings, despite lower retrieval accuracy compared to LSTMs?

Key findings

• The model achieves state-of-the-art performance on Flickr8K and Flickr30K without using object detections, matching or surpassing prior SOTA results.
• With features from a 19-layer Oxford convolutional network, the model sets new SOTA results on both Flickr8K and Flickr30K, demonstrating improved feature utilization.
• The linear encoder enables vector arithmetic in the multimodal space, such that *image of a blue car* − "blue" + "red" yields representations near images of red cars, confirming linguistic regularities in multimodal embeddings.
• Generated captions from the SC-NLM are qualitatively superior to those from prior methods, with improved fluency and diversity, as shown in comparisons with TreeTalk and nearest-neighbor baselines.
• The model successfully retrieves relevant images and sentences using the encoder, while the decoder generates novel, coherent captions from scratch, validating the dual functionality of the framework.
• The framework naturally supports both retrieval (ranking) and generation tasks, demonstrating the unification of visual-semantic embedding and multimodal language modeling.