[Paper Review] Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Bryan A. Plummer|arXiv (Cornell University)|May 19, 2015

Multimodal Machine Learning Applications64 references56 citations

TL;DR

This paper introduces Flickr30k Entities, a large-scale dataset augmenting the Flickr30k image captioning benchmark with 244,035 coreference chains and 275,775 bounding boxes linking phrases in captions to specific image regions. The authors present a strong baseline for phrase localization using image-text embeddings, object detectors, color classification, and size bias, achieving high accuracy despite limited gains in downstream retrieval tasks, highlighting the challenge of grounding language in visual regions.

ABSTRACT

The Flickr30k dataset has become a standard benchmark for sentence-based image description. This paper presents Flickr30k Entities, which augments the 158k captions from Flickr30k with 244k coreference chains, linking mentions of the same entities across different captions for the same image, and associating them with 276k manually annotated bounding boxes. Such annotations are essential for continued progress in automatic image description and grounded language understanding. They enable us to define a new benchmark for localization of textual entity mentions in an image. We present a strong baseline for this task that combines an image-text embedding, detectors for common objects, a color classifier, and a bias towards selecting larger objects. While our baseline rivals in accuracy more complex state-of-the-art models, we show that its gains cannot be easily parlayed into improvements on such tasks as image-sentence retrieval, thus underlining the limitations of current methods and the need for further research.

Motivation & Objective

• Address the lack of detailed, grounded annotations linking textual phrases to visual regions in image captioning datasets.
• Enable more precise language grounding in vision-language models by providing coreference chains and bounding boxes for entity mentions across captions.
• Establish a new benchmark for phrase localization, a critical step toward compositional visual understanding and grounded language generation.
• Investigate whether improved phrase localization can translate into measurable gains in image-sentence retrieval and caption generation.
• Provide a richly annotated dataset to support research in visual question answering, cross-caption coreference, and visual saliency modeling.

Proposed method

• Crowdsourced annotation pipeline split into two stages: coreference resolution and bounding box drawing, with atomic tasks to ensure quality and efficiency.
• Used image-text embeddings to align visual and linguistic representations, combined with object detectors to localize common entities.
• Incorporated a color classifier to improve localization accuracy for color-specific phrases.
• Applied a bias toward selecting larger bounding boxes to favor more salient and likely correct detections.
• Formulated phrase localization as a retrieval task, using a scoring function that combines image-sentence similarity and region-phrase alignment.
• Proposed a normalization term in the scoring function to handle longer sentences with multiple entities, improving performance on complex phrases.

Experimental results

Research questions

• RQ1 Can a strong baseline model for phrase localization achieve high accuracy using simple, modular components like image-text embeddings and object detectors?
• RQ2 To what extent do improvements in phrase localization translate into gains in image-sentence retrieval performance?
• RQ3 How do coreference chains across multiple captions help in grounding ambiguous or plural mentions to specific image regions?
• RQ4 What are the limitations of current models in distinguishing between plausible but incorrect localizations, especially in complex scenes?
• RQ5 Can region-phrase correspondences improve the robustness of image-sentence retrieval in cases with multiple plausible but incorrect sentences?

Key findings

• The proposed baseline for phrase localization achieves high accuracy, rivaling more complex state-of-the-art models, demonstrating the effectiveness of simple, modular components.
• Despite strong localization performance, the gains do not significantly improve image-sentence retrieval accuracy, indicating a gap between localization and retrieval performance.
• The global image-sentence CCA model often retrieves plausible but incorrect sentences, and the region-phrase model fails to make fine-grained distinctions in complex scenes.
• In cases with ambiguous or semantically similar phrases (e.g., 'man with glasses' vs. 'woman with costume glasses'), the model struggles to disambiguate correct from incorrect localizations.
• The normalization term in the scoring function improves performance on longer sentences with multiple entities, showing its utility in handling compositional phrases.
• Qualitative analysis reveals that even with accurate region-phrase alignments, models still fail to capture global scene context, such as spatial relationships and object roles, which are critical for correct inference.