[Paper Review] ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO

Sanghyuk Chun, Wonjae Kim+3 more|arXiv (Cornell University)|Apr 7, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces the ECCV Caption dataset, which corrects false negatives in MS-COCO by collecting machine- and human-verified image-caption associations using five diverse ITM models in a machine-in-the-loop (MITL) annotation pipeline. The resulting dataset offers 3.6× more image-to-caption positives and 8.5× more caption-to-image positives than MS-COCO, and the authors advocate for mAP@R over Recall@K as a more informative evaluation metric, revealing that model rankings shift significantly when using the new benchmark.

ABSTRACT

Image-Text matching (ITM) is a common task for evaluating the quality of Vision and Language (VL) models. However, existing ITM benchmarks have a significant limitation. They have many missing correspondences, originating from the data construction process itself. For example, a caption is only matched with one image although the caption can be matched with other similar images and vice versa. To correct the massive false negatives, we construct the Extended COCO Validation (ECCV) Caption dataset by supplying the missing associations with machine and human annotators. We employ five state-of-the-art ITM models with diverse properties for our annotation process. Our dataset provides x3.6 positive image-to-caption associations and x8.5 caption-to-image associations compared to the original MS-COCO. We also propose to use an informative ranking-based metric mAP@R, rather than the popular Recall@K (R@K). We re-evaluate the existing 25 VL models on existing and proposed benchmarks. Our findings are that the existing benchmarks, such as COCO 1K R@K, COCO 5K R@K, CxC R@1 are highly correlated with each other, while the rankings change when we shift to the ECCV mAP@R. Lastly, we delve into the effect of the bias introduced by the choice of machine annotator. Source code and dataset are available at https://github.com/naver-ai/eccv-caption

Motivation & Objective

• To address the pervasive issue of false negatives in existing Vision-Language (VL) benchmarks like MS-COCO, where only one image-caption pair is labeled as positive despite multiple valid associations.
• To improve the fairness and informativeness of VL model evaluation by constructing a more comprehensive and accurate benchmark with expanded positive correspondences.
• To investigate the bias introduced by machine annotators in the MITL annotation process and propose strategies to mitigate it through multi-model selection.
• To demonstrate that Recall@K metrics are less informative than ranking-based metrics like mAP@R, especially when false negatives are corrected.

Proposed method

• The authors employ a machine-in-the-loop (MITL) annotation pipeline using five state-of-the-art ITM models—CLIP, ViLT, VSRN, PVSE, and PCME—to pre-select candidate image-caption pairs for human validation, reducing the annotation burden from 76 billion to ~1.3 million pairs.
• Human annotators verify the relevance of machine-selected pairs, ensuring high-quality labeling while maintaining scalability.
• The final ECCV Caption dataset contains 1,261 image queries with 17.9 average positive captions each (vs. 5 in MS-COCO) and 1,332 caption queries with 8.5 average positive images each (vs. 1 in MS-COCO).
• The authors propose mAP@R as a superior evaluation metric to Recall@K, as it better captures ranking quality and is more robust to noisy or partially correct annotations.
• To assess model bias, they define a model bias metric $\mathcal{B}_{\Theta}$ as the average absolute performance difference between a model’s score on a subset of MITL models and its score on the full multi-model reference dataset.
• They analyze self-bias and non-self-bias to show that using a single model for MITL leads to high bias, while using multiple models reduces overall bias and improves fairness.

Experimental results

Research questions

• RQ1 How does correcting false negatives in MS-COCO via human-verified, machine-suggested image-caption associations affect the evaluation of Vision-Language models?
• RQ2 To what extent do different machine annotators introduce bias into the MITL annotation process, and how can this bias be quantified and mitigated?
• RQ3 Does the use of mAP@R as an evaluation metric yield more reliable and informative model rankings compared to Recall@K, especially in the presence of corrected positive associations?
• RQ4 Can the rankings of VL models be arbitrarily shifted by changing the underlying machine annotator in the MITL pipeline, and what does this imply for benchmark fairness?
• RQ5 How does the number of machine annotators used in the MITL process affect the overall bias and reliability of the resulting dataset?

Key findings

• The ECCV Caption dataset contains 3.6× more image-to-caption positive associations and 8.5× more caption-to-image positive associations than the original MS-COCO dataset.
• The rankings of 25 pre-trained VL models shift significantly when evaluated on the new mAP@R metric compared to the traditional Recall@K, indicating that prior benchmarks may have masked true model performance differences.
• Using a single machine annotator in the MITL pipeline introduces substantial bias—measured as a model bias $\mathcal{B}_{\Theta}$ of up to 9.5—while using multiple models reduces this bias, with the general bias decreasing as more models are included.
• The self-bias of individual models is low (e.g., 0.1 for PVSE), but the non-self-bias is significantly higher (e.g., 11.8 for PVSE), indicating that a model's performance is more strongly affected by the choice of other models in the MITL pipeline.
• Despite some noisy annotations (e.g., wrong object, color, or quantity), mAP@R is more robust than Recall@K, as it mitigates the impact of false positives by considering ranking quality rather than just top-K correctness.
• The study shows that the choice of machine annotator critically influences dataset quality and model evaluation, and that multi-model MITL strategies are essential for minimizing bias and ensuring fair benchmarking.