[Paper Review] Integrating Image Captioning with Rule-based Entity Masking

Aditya Mogadala, Xiaoyu Shen+1 more|arXiv (Cornell University)|Jul 22, 2020

Multimodal Machine Learning Applications23 references4 citations

TL;DR

This paper proposes Interpret-IC, a novel end-to-end image captioning model that integrates rule-based entity masking using knowledge graph entities to improve interpretability and control over caption generation. By replacing local object features with semantically enriched entity representations and applying human-interpretable masks, the model achieves better object coverage, caption accuracy, and diversity compared to baseline methods, particularly in correcting erroneous object attention.

ABSTRACT

Given an image, generating its natural language description (i.e., caption) is a well studied problem. Approaches proposed to address this problem usually rely on image features that are difficult to interpret. Particularly, these image features are subdivided into global and local features, where global features are extracted from the global representation of the image, while local features are extracted from the objects detected locally in an image. Although, local features extract rich visual information from the image, existing models generate captions in a blackbox manner and humans have difficulty interpreting which local objects the caption is aimed to represent. Hence in this paper, we propose a novel framework for the image captioning with an explicit object (e.g., knowledge graph entity) selection process while still maintaining its end-to-end training ability. The model first explicitly selects which local entities to include in the caption according to a human-interpretable mask, then generate proper captions by attending to selected entities. Experiments conducted on the MSCOCO dataset demonstrate that our method achieves good performance in terms of the caption quality and diversity with a more interpretable generating process than previous counterparts.

Motivation & Objective

• To address the lack of interpretability in existing image captioning models that rely on black-box attention over local image features.
• To enable explicit, human-understandable control over which visual entities are included in the caption through rule-based masking.
• To improve caption quality and diversity by grounding local objects in semantically enriched knowledge graph entities.
• To demonstrate that interpretability in caption generation can enhance both accuracy and diversity without sacrificing end-to-end training capability.

Proposed method

• The model uses a multi-label image classifier to map detected local objects to knowledge graph entities, replacing raw visual features with distributed entity representations.
• A human-interpretable mask is generated via hand-crafted rules that select relevant entities based on semantic relevance and user intent.
• The mask dynamically assigns attention weights to selected entities during caption generation, ensuring focused attention on desired visual objects.
• The framework maintains end-to-end differentiability by integrating the mask as a learnable, soft gating mechanism over entity features.
• The model employs soft attention over masked entity features to generate captions, with the mask acting as a control mechanism for object selection.
• The method is evaluated on MSCOCO using standard metrics including BLEU, ROUGE, and diversity measures (vocab size, novel unigrams/bigrams).

Experimental results

Research questions

• RQ1 Can rule-based entity masking improve the interpretability of image captioning models by making object selection explicit and human-understandable?
• RQ2 Does grounding local objects in knowledge graph entities lead to better caption quality and more accurate object attribution compared to raw visual features?
• RQ3 Can interpretability through masking enhance caption diversity without requiring adversarial or autoencoding training schemes?
• RQ4 To what extent can the mask correct errors made by baseline models in attending to incorrect or irrelevant visual objects?

Key findings

• Interpret-IC achieved a 51.54 score on the novel caption (NC) metric, significantly outperforming the base model (36.23) and approaching the performance of adversarial diversity baselines (68.62).
• The model demonstrated higher unique unigram and bigram diversity across word positions, indicating more varied and less repetitive caption generation.
• Interpret-IC corrected attention errors in the base model, such as incorrectly attending to 'sheep' instead of 'duck', by assigning lower weights to misclassified entities.
• The mask improved object coverage by assigning higher attention weights to prominent and relevant entities, leading to more accurate and informative captions.
• Despite not being trained for diversity, Interpret-IC achieved a 862 vocab size (VS), surpassing the base model (443), indicating improved lexical variety.
• The model maintained strong performance on standard metrics like BLEU and ROUGE, confirming that interpretability does not compromise caption quality.