[Paper Review] Quality Estimation for Image Captions Based on Large-scale Human Evaluations

Tomer Levinboim, Ashish Thapliyal+2 more|arXiv (Cornell University)|Sep 8, 2019

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes a quality estimation (QE) framework for image captions using large-scale human judgments, collecting ~600k binary ratings across 65k image-caption pairs from the Open Images Dataset. The method trains multimodal models on this data to predict caption quality without ground-truth references, achieving high AUC (0.84) in filtering low-quality captions on out-of-domain data, demonstrating strong generalization and practical utility for improving accessibility.

ABSTRACT

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.

Motivation & Objective

• To develop a scalable human evaluation process for collecting coarse-grained caption quality ratings without ground-truth references.
• To create a large-scale, publicly available dataset (Caption-Quality) of ~600k binary human ratings for image caption quality.
• To establish baseline models for quality estimation that generalize to unseen images and outperform simple image-text similarity baselines.
• To validate that QE models trained on coarse ratings can effectively detect and filter low-quality captions using fine-grained human annotations.
• To demonstrate the practical utility of QE models in improving user experience by filtering low-quality captions in real-world deployment.

Proposed method

• Design a crowdsourcing pipeline where annotators rate captions as 'good' or 'bad' based on image context alone, without access to ground-truth captions.
• Collect approximately 600,000 binary quality ratings across 16,000 unique images and 55,000 unique image-caption pairs from the Open Images Dataset.
• Train multimodal neural networks using a bilinear interaction layer to fuse image and caption features for quality estimation.
• Fine-tune the models on a bi-modal pretraining objective using large-scale image-caption datasets like Conceptual Captions to improve generalization.
• Use precision-recall curves and AUC scores to evaluate QE model performance on a held-out dataset with fine-grained annotations from trained raters.
• Apply thresholding on QE scores to filter out low-quality captions, with evaluation based on majority agreement on correctness and helpfulness from expert raters.

Experimental results

Research questions

• RQ1 Can a scalable human evaluation process collect consistent and reliable quality signals for image captions without ground-truth references?
• RQ2 Is the quality signal encoded in coarse binary ratings learnable by neural models and generalizable to out-of-domain images?
• RQ3 Can QE models trained on coarse ratings effectively distinguish between correct-and-helpful and incorrect-or-unhelpful captions on a fine-grained evaluation set?
• RQ4 How does bi-modal pretraining improve the performance of quality estimation models on this task?
• RQ5 To what extent can QE models improve real-world captioning systems by filtering low-quality outputs?

Key findings

• The Caption-Quality dataset contains a stable and consistent signal, with ~600,000 human ratings across 65k image-caption pairs collected via a scalable crowdsourcing process.
• The best-performing QE model achieved a Spearman correlation of 0.72 on the dev set and 0.70 on the test set, significantly outperforming image-text similarity baselines.
• Pretraining on large-scale image-caption data improved QE model performance, with fine-tuned models achieving an AUC of 0.84 on the extrinsic evaluation set.
• The pretrained and fine-tuned bilinear model achieved a precision of 0.8 at a recall of 0.71, representing a 3.4x improvement in coverage over the image-text similarity baseline.
• QE models trained on coarse ratings successfully generalized to filter low-quality captions on out-of-domain images, even without exposure to fine-grained annotations during training.
• The dataset enables effective downstream applications such as caption re-ranking and reinforcement learning-based caption generation, as demonstrated by its utility in extrinsic evaluations.