[Paper Review] Microsoft COCO Captions: Data Collection and Evaluation Server

Xinlei Chen, Hao Fang+5 more|arXiv (Cornell University)|Apr 1, 2015

Multimodal Machine Learning Applications47 references1,628 citations

TL;DR

This paper introduces the Microsoft COCO Captions dataset, comprising over 1.5 million human-annotated captions for 330,000 images, and a standardized evaluation server that scores automatic captioning models using BLEU, METEOR, ROUGE, and CIDEr metrics. The key contribution is a consistent, scalable evaluation framework with human-annotated references and an online server to enable fair, reproducible benchmarking of image captioning algorithms across training, validation, and test sets.

ABSTRACT

In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

Motivation & Objective

• To establish a large-scale, human-annotated image caption dataset for training and evaluating image captioning models.
• To address inconsistencies in automatic evaluation by providing a centralized, standardized evaluation server with consistent metric implementations.
• To improve correlation between automatic metrics and human judgment by using multiple reference captions per image, especially in the c40 subset with 40 references per image.
• To enable fair comparison of caption generation models by decoupling test set evaluation from public reference captions, reducing overfitting risk.
• To support future human evaluation studies to validate and refine automatic metrics against human judgment.

Proposed method

• Data collection via Amazon Mechanical Turk using standardized instructions to ensure descriptive, contextually relevant captions with at least 8 words and no pronouns or future/past tense.
• Creation of two datasets: MS COCO c5 (5 references per image) and MS COCO c40 (40 references for 5,000 test images) to improve metric correlation with human judgment.
• Implementation of a centralized evaluation server hosted on CodaLab that computes BLEU-1 to BLEU-4, METEOR, ROUGE-L, and CIDEr-D scores on submitted candidate captions.
• Use of Stanford PTBTokenizer for consistent tokenization and preprocessing of both candidate and reference captions.
• Introduction of a novel metric, Precision at Human Recall (PHR), to evaluate system performance relative to human agreement patterns.
• Modeling of human captioning behavior using a probabilistic framework to estimate word usage probability (p) and fixed recall, enabling analysis of precision-recall dynamics with varying numbers of reference captions.

Experimental results

Research questions

• RQ1 How can a large-scale, diverse, and consistent image caption dataset be collected at scale using human annotators?
• RQ2 To what extent do automatic evaluation metrics correlate with human judgment, and how can this correlation be improved?
• RQ3 What is the impact of increasing the number of reference captions per image on the reliability and fairness of automatic evaluation metrics?
• RQ4 How can a standardized, centralized evaluation server reduce variability and overfitting in benchmarking image captioning models?
• RQ5 Can a new metric like Precision at Human Recall (PHR) better reflect system performance relative to human captioning behavior?

Key findings

• The final dataset contains 1,026,459 captions across 330,000 images, with 5 references per image in c5 and 40 in c40, collected via Mechanical Turk with strict annotation guidelines.
• The evaluation server computes multiple standard metrics—BLEU, METEOR, ROUGE, and CIDEr—on submitted candidate captions, enabling consistent, reproducible benchmarking.
• Using 40 reference captions per image (c40) significantly improves the correlation between automatic metrics and human judgment compared to using only 5 references.
• Human agreement precision decreases with fewer reference captions, but recall remains constant, validating the use of a fixed recall model to estimate human behavior.
• The proposed PHR metric shows strong alignment with human precision at fixed recall, offering a more reliable benchmark for system performance than standard metrics alone.
• The model predicts that human precision approaches 1.0 as the number of annotators increases, confirming that with sufficient annotations, word usage becomes highly consistent for visible objects.