[Paper Review] Image Captioning with Semantic Attention

Quanzeng You, Hailin Jin+3 more|arXiv (Cornell University)|Mar 12, 2016

Multimodal Machine Learning Applications34 references247 citations

TL;DR

Proposes a semantic attention model that fuses top-down CNN features with bottom-up detected visual concepts inside an RNN to generate image captions, achieving state-of-the-art results on MS-COCO and Flickr30K.

ABSTRACT

Automatically generating a natural language description of an image has attracted interests recently both because of its importance in practical applications and because it connects two major artificial intelligence fields: computer vision and natural language processing. Existing approaches are either top-down, which start from a gist of an image and convert it into words, or bottom-up, which come up with words describing various aspects of an image and then combine them. In this paper, we propose a new algorithm that combines both approaches through a model of semantic attention. Our algorithm learns to selectively attend to semantic concept proposals and fuse them into hidden states and outputs of recurrent neural networks. The selection and fusion form a feedback connecting the top-down and bottom-up computation. We evaluate our algorithm on two public benchmarks: Microsoft COCO and Flickr30K. Experimental results show that our algorithm significantly outperforms the state-of-the-art approaches consistently across different evaluation metrics.

Motivation & Objective

• Motivate and address the gap between top-down and bottom-up captioning approaches by leveraging semantic attention.
• Develop a model that attends to semantically meaningful concepts and fuses them with global image features during caption generation.
• Demonstrate improved caption quality on standard benchmarks and analyze attention behavior and attribute predictions.

Proposed method

• Extract both global visual features from a CNN and a set of detected visual attributes (A_i) from the image.
• Use an LSTM/RNN to generate captions, with an input attention mechanism (alpha_t^i) selecting attributes conditioned on the previous word.
• Incorporate an output attention mechanism (beta_t^i) conditioning the word prediction on attended attributes and the current RNN state.
• Compute attention scores via bilinear/embedding-based projections to produce a weighted sum of attribute embeddings for input and output, integrating with the recurrent state.
• Train end-to-end with a negative log-likelihood objective plus regularization terms that promote complete and sparse attention across attributes (g(alpha), g(beta)).
• Predict attributes via non-parametric (k-NN using weakly labeled images) and parametric methods (ranking loss multi-label classifier and fully convolutional networks).

Experimental results

Research questions

• RQ1 Can semantic attention over detected visual concepts improve image captioning beyond purely top-down or bottom-up approaches?
• RQ2 How should input and output attention over attributes be designed to best influence RNN state updates and word predictions?
• RQ3 What is the impact of using different attribute prediction strategies (k-NN, RK, FCN) on caption quality?
• RQ4 Does combining global features with semantically attended attributes yield better results across standard metrics (BLEU, METEOR, ROUGE-L, CIDEr)?

Key findings

• The semantic attention model significantly outperforms state-of-the-art methods on MS-COCO and Flickr30K across multiple metrics.
• Attention on visual attributes via both input and output mechanisms improves caption quality, with the combination yielding the best results.
• FCN-based attribute predictions tend to provide more robust attributes than ranking-loss or k-NN approaches in their impact on captioning performance.
• Using the top 3 attended attributes often yields the best performance among fusion strategies, compared to simple max or concatenation.
• Ground-truth visual attributes provide an upper bound and show substantial potential gains, indicating that attribute quality strongly affects caption quality.