[Paper Review] End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Pranava Madhyastha, Josiah Wang+1 more|arXiv (Cornell University)|Sep 11, 2018

Multimodal Machine Learning Applications27 references3 citations

TL;DR

This paper investigates whether end-to-end image captioning models generate captions by exploiting distributional similarity in a multimodal feature space rather than true visual understanding. By varying image representations while keeping the RNN decoder fixed, the authors demonstrate that models robustly match test images to semantically similar training images and generate captions based on these matches, even with noisy or compressed inputs, indicating they rely on similarity matching over deep visual understanding.

ABSTRACT

We hypothesize that end-to-end neural image captioning systems work seemingly well because they exploit and learn `distributional similarity' in a multimodal feature space by mapping a test image to similar training images in this space and generating a caption from the same space. To validate our hypothesis, we focus on the `image' side of image captioning, and vary the input image representation but keep the RNN text generation component of a CNN-RNN model constant. Our analysis indicates that image captioning models (i) are capable of separating structure from noisy input representations; (ii) suffer virtually no significant performance loss when a high dimensional representation is compressed to a lower dimensional space; (iii) cluster images with similar visual and linguistic information together. Our findings indicate that our distributional similarity hypothesis holds. We conclude that regardless of the image representation used image captioning systems seem to match images and generate captions in a learned joint image-text semantic subspace.

Motivation & Objective

• To challenge the assumption that end-to-end image captioning models achieve strong performance through true visual and linguistic understanding.
• To investigate whether image captioning systems rely on distributional similarity in a joint visual-semantic space rather than semantic inference.
• To evaluate the robustness of captioning models to variations in image representation, including noise and dimensionality reduction.
• To examine the domain generalization capabilities of captioning models under distribution shift between training and test data.
• To provide empirical evidence that caption generation is driven by image matching in a learned visual-semantic subspace.

Proposed method

• The authors vary the input image representation (e.g., ResNet-152 features, sparse BoW, pseudo-random vectors) while keeping the RNN-based text generation component fixed.
• They evaluate model performance using standard automatic metrics (BLEU, METEOR, CIDEr, SPICE) across different image representations.
• They introduce pseudo-random vectors derived from object-level features to test the model’s ability to separate meaningful structure from noise.
• They compress high-dimensional image embeddings into lower-dimensional spaces to assess performance degradation under dimensionality reduction.
• They evaluate cross-domain generalization by testing models trained on MSCOCO on the Flickr30k test set.
• They analyze clustering behavior of image representations in the visual-semantic embedding space to assess semantic grouping.

Experimental results

Research questions

• RQ1 Do end-to-end image captioning models generate captions by matching test images to similar training images in a multimodal embedding space?
• RQ2 Can models maintain strong performance when fed noisy or random image representations, indicating robustness to input structure?
• RQ3 To what extent does performance degrade when high-dimensional image features are compressed into lower-dimensional representations?
• RQ4 How does model performance change when test data comes from a different distribution than the training data?
• RQ5 Do image representations cluster in the visual-semantic space according to shared visual and linguistic content?

Key findings

• End-to-end image captioning models are capable of separating meaningful structure from noisy pseudo-random image representations, indicating robustness to input perturbations.
• Performance remains virtually unchanged when high-dimensional image embeddings are compressed into lower-dimensional spaces, supporting the hypothesis of similarity-based matching over semantic understanding.
• Images with similar visual and linguistic content cluster together in the learned visual-semantic subspace, confirming the existence of a shared semantic embedding space.
• Models trained on MSCOCO show a significant drop in performance when evaluated on Flickr30k, despite only 8.6% out-of-vocabulary word overlap, indicating strong domain dependency.
• The drop in BLEU and METEOR scores on Flickr30k is primarily due to structural and length differences in captions, not vocabulary mismatch, suggesting that models fail to generalize across linguistic distributions.
• The results collectively support the hypothesis that end-to-end captioning models exploit distributional similarity in a multimodal space rather than performing deep visual or linguistic reasoning.