[Paper Review] Image Captioning: Transforming Objects into Words

Simao Herdade, Armin Kappeler+2 more|arXiv (Cornell University)|Jun 14, 2019

Multimodal Machine Learning ApplicationsComputer Science26 references93 citations

TL;DR

Introduces the Object Relation Transformer, which incorporates geometric attention over spatial relationships of detected objects to improve image captioning on MS-COCO, achieving state-of-the-art results among single-model approaches.

ABSTRACT

Image captioning models typically follow an encoder-decoder architecture which uses abstract image feature vectors as input to the encoder. One of the most successful algorithms uses feature vectors extracted from the region proposals obtained from an object detector. In this work we introduce the Object Relation Transformer, that builds upon this approach by explicitly incorporating information about the spatial relationship between input detected objects through geometric attention. Quantitative and qualitative results demonstrate the importance of such geometric attention for image captioning, leading to improvements on all common captioning metrics on the MS-COCO dataset.

Motivation & Objective

• Motivate and improve image captioning by explicitly modeling spatial relationships between detected objects.
• Integrate geometric attention into a Transformer-based encoder for caption generation.
• Demonstrate quantitative and qualitative gains on MS-COCO over baselines and prior methods.

Proposed method

• Use Faster R-CNN (ResNet-101) to detect objects and extract 2048-dim features per box.
• Replace standard Transformer encoder attention with a combined appearance+geometric attention in which geometric weights are derived from relative box positions and sizes.
• Compute relative geometry lambda(m,n) and embed it to produce omega_G, then form combined attention omega^{mn} = (omega_G^{mn} exp(omega_A^{mn})) / sum_l omega_G^{ml} exp(omega_A^{ml}).
• Train with cross-entropy then fine-tune with self-critical sequence training (CIDEr-D optimization) and beam search.
• Evaluate on MS-COCO 2014 Captions using CIDEr-D, SPICE, BLEU, METEOR, and ROUGE-L metrics.

Experimental results

Research questions

• RQ1 Does incorporating spatial relationships between detected objects via geometric attention improve image captioning performance?
• RQ2 How does Object Relation Transformer compare to standard Transformer and to strong baselines on MS-COCO?
• RQ3 What is the impact of geometric attention on SPICE subcategories related to relations and counts?

Key findings

• Object Relation Transformer improves CIDEr-D, SPICE, BLEU-1, BLEU-4, METEOR, and ROUGE-L over the Standard Transformer, with statistically significant gains on several metrics.
• Geometric attention yields higher SPICE Relation and Count scores, indicating better relational reasoning and object counting in captions.
• Ablation shows adding object relations to Transformer produces larger gains in CIDEr-D and BLEU metrics, especially with beam search.
• Compared to size-based or left-right/top-bottom orderings, geometric attention improves CIDEr-D (112.6 vs 111.0 and variations) demonstrating its effectiveness over simple positional encodings.
• Qualitative examples show improved spatial awareness and more accurate relations (e.g., “two chairs under an umbrella”).
• Significant improvements in the SPICE Count subcategory from 11.30 to 17.51 when using geometric attention.