[Paper Review] Retrieval-augmented Image Captioning

Rita Ramos, Desmond Elliott+1 more|arXiv (Cornell University)|Feb 16, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes extra, a retrieval-augmented image captioning model that improves caption generation by jointly encoding the input image and k retrieved captions using a pretrained vision-and-language BERT (LXMERT). By leveraging cross-modal representations from both visual regions and semantically relevant textual context, the model achieves state-of-the-art performance on COCO without retraining when using external data, demonstrating that retrieval-augmented generation enhances caption quality through richer multimodal context.

ABSTRACT

Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore, as opposed to the image alone. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT, while the decoder attends to the multimodal encoder representations, benefiting from the extra textual evidence from the retrieved captions. Experimental results on the COCO dataset show that image captioning can be effectively formulated from this new perspective. Our model, named EXTRA, benefits from using captions retrieved from the training dataset, and it can also benefit from using an external dataset without the need for retraining. Ablation studies show that retrieving a sufficient number of captions (e.g., k=5) can improve captioning quality. Our work contributes towards using pretrained V&L encoders for generative tasks, instead of standard classification tasks.

Motivation & Objective

• To address the limitation of standard image captioning models that rely solely on visual features by incorporating external textual context.
• To explore whether pretrained vision-and-language BERTs can be effectively repurposed for generative tasks beyond classification.
• To evaluate the impact of retrieving multiple captions from a datastore on captioning performance.
• To enable zero-shot adaptation to external datasets without retraining the main model.
• To improve interpretability and reduce bias by making retrieved captions visible during inference.

Proposed method

• The model uses a pretrained LXMERT encoder to jointly process the input image (represented as 36 object regions) and k retrieved captions.
• Retrieved captions are tokenized and concatenated with special CLS and SEP tokens, forming a single input sequence for the V&L encoder.
• The retrieval system uses FAISS to find k nearest captions based on image embedding similarity, using a datastore of image-caption pairs.
• A Transformer decoder attends to the cross-modal encoder outputs to generate the final caption.
• The model is fine-tuned on COCO using standard language modeling objectives, with no retraining needed when using external data.
• The approach enables in-domain and out-of-domain generalization by dynamically retrieving relevant captions from external datasets.

Experimental results

Research questions

• RQ1 Can joint encoding of images and retrieved captions using a pretrained V&L BERT improve image captioning performance?
• RQ2 How does the number of retrieved captions (k) affect caption quality and model robustness?
• RQ3 Can the model benefit from retrieving captions from an external dataset without retraining?
• RQ4 Does the use of retrieved captions improve interpretability and reduce reliance on visual attention maps alone?
• RQ5 How do biases in the datastore influence the generated captions, and can retrieval help mitigate or reveal such biases?

Key findings

• The model achieves competitive performance on the COCO dataset, outperforming standard image-only captioning baselines when using retrieved captions.
• Ablation studies show that retrieving k=5 captions yields the best performance, indicating that multiple retrieved examples provide richer context.
• The model can effectively utilize captions from an external dataset (e.g., COCO) to improve performance on Flickr30K without retraining, demonstrating zero-shot generalization.
• The model’s performance improves when the retrieved captions are semantically relevant, suggesting that retrieval quality directly impacts caption quality.
• Mismatched or biased retrieved captions can lead to incorrect or biased outputs, highlighting the importance of analyzing retrieved examples for interpretability.
• The model provides interpretability through visible retrieved captions, offering a transparent alternative to black-box attention maps.