[Paper Review] Language Models for Image Captioning: The Quirks and What Works

Jacob Devlin, Hao Cheng+6 more|arXiv (Cornell University)|May 7, 2015

Multimodal Machine Learning Applications23 references119 citations

TL;DR

This paper compares language modeling approaches for image captioning using the same state-of-the-art CNN features, finding that while a Multimodal Recurrent Neural Network (MRNN) achieves higher BLEU scores, a two-stage Maximum Entropy Language Model (ME LM) with Deep Multimodal Similarity Model (DMSM) re-ranking generates more novel, human-preferred captions. The ME LM + DMSM system sets a new SOTA BLEU score on COCO, but human evaluations show it outperforms automatic metrics, highlighting a disconnect between automatic and human judgments.

ABSTRACT

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.

Motivation & Objective

• To compare the effectiveness of different language modeling approaches—specifically ME LM and RNN—when conditioned on the same state-of-the-art CNN features for image captioning.
• To investigate why automatic metrics like BLEU may not align with human judgments in image captioning.
• To evaluate the role of caption novelty and data set diversity in model performance, particularly on compositionally novel images.
• To assess the impact of data set overlap and linguistic irregularities on caption generation quality.
• To determine whether simple retrieval methods like k-nearest neighbor can rival complex neural models in performance.

Proposed method

• The authors use a 16-layer VGGNet fine-tuned on COCO as a shared visual encoder to extract features for all models, ensuring consistent input across comparisons.
• For the ME LM approach, a bag of words is extracted from the CNN features above a threshold (α=0.5), and a beam search generates sentences covering a subset of these words, with dynamic pruning of already-generated words.
• The D-ME+DMSM system uses a Deep Multimodal Similarity Model to re-rank the n-best outputs from the ME LM, improving semantic relevance by matching image and text embeddings in a shared vector space.
• The MRNN model directly conditions a gated recurrent neural network (GRNN) on the final CNN activation (fc7 layer), generating captions autoregressively without discrete word detection.
• The k-nearest neighbor baseline retrieves the most similar caption from the training set based on image feature similarity, serving as a strong baseline for comparison.
• Performance is evaluated using BLEU scores on the COCO test set, human judgments on caption quality, and analysis of caption novelty and repetition.

Experimental results

Research questions

• RQ1 How do ME language models and RNN-based models compare when conditioned on identical CNN features for image captioning?
• RQ2 Why does the MRNN achieve higher BLEU scores than the ME LM, despite generating more repeated and less novel captions?
• RQ3 To what extent does data set overlap and caption repetition affect model generalization and human evaluation outcomes?
• RQ4 Can a simple k-nearest neighbor retrieval method perform competitively with complex neural captioning models on the COCO dataset?
• RQ5 Does incorporating a DMSM for re-ranking significantly improve caption quality beyond automatic metrics like BLEU?

Key findings

• The MRNN model achieves a BLEU score of 25.7 on the full COCO test set, outperforming the ME LM and D-ME+DMSM systems in automatic metrics.
• The D-ME+DMSM system achieves a new SOTA BLEU score of 27.3 when combined with the MRNN, surpassing previously published results by 1.6 BLEU points.
• Despite higher BLEU scores, the MRNN generates significantly more repeated captions, with 60.3% of its outputs seen in the training set, compared to 30.0% for D-ME+DMSM.
• The D-ME+DMSM system produces 47.0% unique captions, indicating greater novelty, and outperforms the MRNN by 2.1 BLEU points on the 20% least-overlapping (compositionally novel) images.
• Human evaluations show that D-ME+DMSM is significantly preferred over the MRNN, indicating that human quality judgments favor novelty and generalization over high BLEU scores.
• The k-nearest neighbor method performs similarly to the MRNN in both BLEU (26.0) and human judgments, suggesting that retrieval-based systems can be highly competitive.