[Paper Review] Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning

Shikha Dubey, Farrukh Olimov+3 more|arXiv (Cornell University)|Sep 16, 2021

Multimodal Machine Learning Applications44 references4 citations

TL;DR

This paper proposes LATGeO, a label-attention Transformer for image captioning that enhances caption quality by integrating geometrically coherent object proposals and a novel label-attention module. By leveraging object detection, localized geometric ratios, and label-aware attention, LATGeO achieves state-of-the-art performance on MSCOCO, outperforming existing methods with a CIDEr score of 131.7 and BLEU-1 of 81.0.

ABSTRACT

Automatic transcription of scene understanding in images and videos is a step towards artificial general intelligence. Image captioning is a nomenclature for describing meaningful information in an image using computer vision techniques. Automated image captioning techniques utilize encoder and decoder architecture, where the encoder extracts features from an image and the decoder generates a transcript. In this work, we investigate two unexplored ideas for image captioning using transformers: First, we demonstrate the enforcement of using objects' relevance in the surrounding environment. Second, learning an explicit association between labels and language constructs. We propose label-attention Transformer with geometrically coherent objects (LATGeO). The proposed technique acquires a proposal of geometrically coherent objects using a deep neural network (DNN) and generates captions by investigating their relationships using a label-attention module. Object coherence is defined using the localized ratio of the geometrical properties of the proposals. The label-attention module associates the extracted objects classes to the available dictionary using self-attention layers. The experimentation results show that objects' relevance in surroundings and binding of their visual feature with their geometrically localized ratios combined with its associated labels help in defining meaningful captions. The proposed framework is tested on the MSCOCO dataset, and a thorough evaluation resulting in overall better quantitative scores pronounces its superiority.

Motivation & Objective

• Address the lack of explicit modeling of object relationships and visual-semantic alignment in existing image captioning models.
• Improve caption quality by incorporating geometric coherence—defined by localized ratios of object dimensions—into the attention mechanism.
• Bridge the visual and language modalities by introducing a label-attention module that associates detected object classes with linguistic constructs during decoding.
• Demonstrate that combining object-level features, spatial geometry, and label-aware attention leads to more semantically accurate and contextually relevant captions.
• Achieve superior performance on the MS COCO benchmark using a single-model architecture without ensemble methods.

Proposed method

• Use Faster R-CNN to extract object proposals as visual inputs, providing fine-grained region-level features.
• Compute geometric coherence by measuring the localized ratio of object dimensions (e.g., height/width or relative size) to encode spatial relationships.
• Introduce a label-attention module (LAM) that injects object class labels as prior information into each decoder layer via self-attention, enhancing semantic grounding.
• Employ a fully connected encoder-decoder transformer architecture, where all encoder outputs are connected to all decoder inputs to enable multi-level feature fusion.
• Train the model using cross-entropy loss and fine-tune with reinforcement learning (PPO) to optimize dense evaluation metrics like CIDEr and SPICE.
• Utilize residual and skip connections in the encoder and decoder stacks to stabilize training and improve gradient flow.

Experimental results

Research questions

• RQ1 Can geometrically coherent object proposals—defined by localized ratios of object dimensions—improve the spatial reasoning of image captioning models?
• RQ2 Does explicitly associating object labels with decoder attention improve the alignment between visual content and generated language?
• RQ3 How does a fully connected encoder-decoder transformer architecture compare to standard stack-based or skip-connection variants in image captioning?
• RQ4 Can a single-model LATGeO framework outperform multi-model ensembles in terms of both automatic and human evaluation metrics?
• RQ5 What is the optimal configuration of encoder-decoder layer connectivity and depth for achieving state-of-the-art performance?

Key findings

• LATGeO achieves a CIDEr score of 131.7 on the MS COCO test set, outperforming previous SOTA methods using a single model.
• The model achieves a BLEU-1 score of 81.0, indicating strong performance in n-gram overlap with reference captions.
• The fully connected encoder-decoder architecture with 3 layers and residual connections in both encoder and decoder layers yields the best performance.
• Ablation studies confirm that the label-attention module significantly improves performance, particularly in SPICE (22.9) and ROUGE-L (58.7) metrics.
• Fine-tuning with reinforcement learning further boosts CIDEr and SPICE scores, demonstrating the effectiveness of online optimization.
• The use of geometrically coherent object proposals leads to more accurate spatial descriptions, such as 'in front of' or 'next to', improving contextual realism.