[Paper Review] Technical Report: Image Captioning with Semantically Similar Images

Martin Kolář, Michal Hradiš+1 more|arXiv (Cornell University)|Jun 12, 2015

Multimodal Machine Learning Applications5 references5 citations

TL;DR

This paper proposes a simple yet effective image captioning method that uses CNN embeddings to retrieve semantically similar training images and selects the most representative caption based on unigram frequency from their collective captions. Despite low scores on standard automatic metrics, it outperforms human captions in Turing test pass rate and is competitive in human evaluation for being judged as better or equal.

ABSTRACT

This report presents our submission to the MS COCO Captioning Challenge 2015. The method uses Convolutional Neural Network activations as an embedding to find semantically similar images. From these images, the most typical caption is selected based on unigram frequencies. Although the method received low scores with automated evaluation metrics and in human assessed average correctness, it is competitive in the ratio of captions which pass the Turing test and which are assessed as better or equal to human captions.

Motivation & Objective

• To develop a simple, non-generative image captioning approach that leverages semantic similarity for caption retrieval.
• To improve caption quality by aggregating and selecting from multiple captions of semantically similar training images.
• To evaluate the method's performance not only on standard metrics but also on human judgment and Turing test criteria.
• To establish a strong baseline for image captioning using minimal architectural complexity and no end-to-end training.

Proposed method

• Utilizes the last fully connected layer of a pre-trained Caffe CNN (ImageNet-ILSVRC) as a 4096-dimensional semantic embedding for images.
• Finds the 10 nearest training images to a test image using cosine distance in the CNN embedding space.
• Collects all 50 captions (5 per image × 10 nearest images) into a candidate corpus, ignoring image ranking.
• Applies a frequency-based culling process: iteratively removes sentences that do not contain the most frequent word in the corpus, skipping the 100 most common words from Google N-grams.
• Continues the culling process until only one sentence remains, which is selected as the final caption.
• Employs a fixed n=10 for nearest neighbor search, with cosine distance outperforming other distance metrics.

Experimental results

Research questions

• RQ1 Can a simple, non-generative method based on semantic similarity and unigram frequency outperform more complex models in human evaluation?
• RQ2 How does caption quality based on unigram frequency compare to human-annotated captions in terms of naturalness and coherence?
• RQ3 To what extent can a retrieval-based captioning system pass the Turing test compared to human-generated captions?
• RQ4 Does the method achieve better performance in human judgment for 'better or equal' than human captions despite low automatic metric scores?

Key findings

• The method achieved a 19.4% ratio of captions judged as better or equal to human captions, significantly outperforming random (0.7%) and approaching human performance (63.8%).
• 21.3% of the generated captions passed the Turing test, compared to 67.5% for human captions and only 2.0% for random captions.
• The average correctness score was 3.079 out of 5, which is substantially higher than random (1.084) but lower than human (4.836).
• The average detail level was 3.482 out of 5, slightly higher than human (3.428) and significantly above random (3.247), indicating decent descriptive quality.
• The method achieved a 15.4% ratio of captions judged as similar to human descriptions, compared to 35.2% for human captions and 1.3% for random.
• Despite low scores on standard automatic metrics, the method demonstrated strong performance in human-centric evaluations, particularly in subjective quality and human-likeness.