[Paper Review] Exploring Nearest Neighbor Approaches for Image Captioning

Jacob Devlin, Saurabh Gupta+3 more|arXiv (Cornell University)|May 17, 2015

Multimodal Machine Learning Applications34 references160 citations

TL;DR

This paper evaluates nearest neighbor (NN) approaches for image captioning by retrieving similar training images and selecting a consensus caption from their captions using CIDEr or BLEU scores. Despite outperforming many neural captioning models on automatic metrics like BLEU and CIDEr, human evaluations show that generative models are still preferred, highlighting a disconnect between automatic metrics and human judgment.

ABSTRACT

We explore a variety of nearest neighbor baseline approaches for image captioning. These approaches find a set of nearest neighbor images in the training set from which a caption may be borrowed for the query image. We select a caption for the query image by finding the caption that best represents the "consensus" of the set of candidate captions gathered from the nearest neighbor images. When measured by automatic evaluation metrics on the MS COCO caption evaluation server, these approaches perform as well as many recent approaches that generate novel captions. However, human studies show that a method that generates novel captions is still preferred over the nearest neighbor approach.

Motivation & Objective

• To investigate whether simple nearest neighbor approaches can match or exceed state-of-the-art neural captioning models on automatic evaluation metrics.
• To assess the role of caption generation versus caption retrieval in achieving human-like image captioning.
• To evaluate the gap between automatic metrics (BLEU, CIDEr) and human judgment in caption quality assessment.
• To explore the effectiveness of different image feature representations—GIST, pre-trained deep features, and fine-tuned features—for nearest neighbor retrieval in captioning.
• To provide a baseline for future research by comparing retrieval-based methods with generation-based models on the MS COCO dataset.

Proposed method

• For each query image, the method retrieves k nearest training images using various feature spaces: GIST, pre-trained ImageNet features (fc7), and fine-tuned features for caption generation.
• Candidate captions are collected from the k nearest neighbor images, forming a set of potential captions for the query image.
• A consensus caption is selected by maximizing a score that measures agreement with other candidate captions, using either CIDEr or BLEU as the scoring function.
• The final caption is chosen based on its highest consensus score among the candidate captions, with CIDEr favoring more detailed and diverse captions.
• The approach is evaluated on the MS COCO test set using standard automatic metrics (BLEU, METEOR, CIDEr) and human evaluation via crowdsourcing.
• Human evaluation compares system-generated captions against human-written captions, measuring the percentage of captions judged as better, equal, or worse.

Experimental results

Research questions

• RQ1 Can a nearest neighbor-based captioning approach achieve performance comparable to state-of-the-art neural captioning models on automatic evaluation metrics?
• RQ2 Why do human evaluators still prefer generative captioning models over retrieval-based methods despite similar automatic metric scores?
• RQ3 How do different image feature representations (GIST, pre-trained, fine-tuned) affect the quality of retrieved captions?
• RQ4 To what extent do automatic metrics like BLEU and CIDEr correlate with human judgment in image captioning?
• RQ5 Can a hybrid approach combining nearest neighbor retrieval and neural generation improve overall caption quality?

Key findings

• The nearest neighbor approach using fine-tuned deep features (fc7-fine) achieved second or third place on the MS COCO test set across multiple metrics, including CIDEr and BLEU.
• The fc7-fine model scored 26.5 CIDEr and 25.1 BLEU, outperforming several recent neural captioning models in automatic evaluation.
• Despite strong automatic metric scores, human evaluations showed that only 27.6% of NN-generated captions were judged as better than or equal to human-written captions, compared to 34.0% for a generative model (ME + DMSM).
• The consensus caption selection method using CIDEr produced more detailed and contextually richer captions than the BLEU-based method.
• Fine-tuned deep features (fc7-fine) significantly outperformed GIST and non-fine-tuned features in nearest neighbor retrieval quality.
• The results suggest a significant disconnect between automatic metrics and human judgment, as automatic scores did not predict human preference.