[Paper Review] SuperCaptioning: Image Captioning Using Two-dimensional Word Embedding

Baohua Sun, Lin Yang+5 more|arXiv (Cornell University)|May 25, 2019

Multimodal Machine Learning Applications16 references4 citations

TL;DR

This paper proposes SuperCaptioning, a novel image captioning method that uses two-dimensional word embedding to jointly process visual and textual features in a single CNN model. By converting text into image-like representations and combining them with input images, the method achieves high-quality captions on Flickr30k, outperforming conventional two-stream approaches through end-to-end training on a unified architecture.

ABSTRACT

Language and vision are processed as two different modal in current work for image captioning. However, recent work on Super Characters method shows the effectiveness of two-dimensional word embedding, which converts text classification problem into image classification problem. In this paper, we propose the SuperCaptioning method, which borrows the idea of two-dimensional word embedding from Super Characters method, and processes the information of language and vision together in one single CNN model. The experimental results on Flickr30k data shows the proposed method gives high quality image captions. An interactive demo is ready to show at the workshop.

Motivation & Objective

• To address the limitation of current image captioning models that process vision and language separately.
• To explore the effectiveness of two-dimensional word embedding—originally used in text classification—within image captioning.
• To unify image and text processing into a single CNN model for end-to-end caption generation.
• To enable efficient on-device inference using low-power CNN accelerators.

Proposed method

• The method converts raw text into a 2D image representation using the Squared English Word (SEW) technique, where each character occupies a fixed grid space.
• Input images and text embeddings are combined into a single 224×224 SuperCaptioning image, with the image placed at the top and text at the bottom.
• A pre-trained SE-Net-154 model fine-tuned on ImageNet is used as the backbone, with its final layer reconfigured to 11,571 classes corresponding to the vocabulary size.
• The model predicts the next word in the caption sequence iteratively, with prediction ending when EOS (End of Sentence) token is generated or the maximum word count (14) is reached.
• Training data is generated by labeling each SuperCaptioning image with the next word in the ground-truth caption, treating caption generation as a multi-class classification task.
• The method leverages the Super Characters framework’s success in text classification, adapting it to joint visual-linguistic modeling for image captioning.

Experimental results

Research questions

• RQ1 Can two-dimensional word embedding improve image captioning performance by enabling joint processing of vision and language in a single CNN?
• RQ2 How does the SuperCaptioning method compare to conventional two-stream models in terms of caption quality and coherence?
• RQ3 What hyperparameters (e.g., font size, cut-length, image resizing) yield optimal captioning results on Flickr30k?
• RQ4 Can the method generate descriptive captions that capture object count, color, background context, and action details?
• RQ5 Is the approach suitable for deployment on low-power edge devices with dedicated CNN accelerators?

Key findings

• The SuperCaptioning method generates high-quality image captions that accurately describe object count, as seen in the example of four men in life jackets on a boat.
• The model effectively captures color information, such as a woman in a black coat holding a red umbrella, demonstrating fine-grained visual-linguistic alignment.
• The method generates contextually rich captions, including background elements like mountains, indicating strong scene understanding.
• The model produces detailed action descriptions, such as performers with arms outstretched in a ballet, showing capability in capturing dynamic scenes.
• The approach achieves strong performance on Flickr30k, with 31,333 images retained after filtering for captions under 14 words.
• The method is suitable for on-device deployment due to its reliance on efficient CNN inference, especially with emerging low-power accelerators.