[Paper Review] 3M: Multi-style image caption generation using Multi-modality features under Multi-UPDOWN model

Chengxi Li, Brent Harrison|arXiv (Cornell University)|Mar 20, 2021

Multimodal Machine Learning Applications20 references4 citations

TL;DR

This paper proposes 3M, a multi-style image captioning model that fuses multi-modality features—ResNeXt visual features and DenseCap-generated text features—within a Multi-UPDOWN attention architecture to generate diverse, stylistically controlled captions. The model achieves state-of-the-art performance on FlickrStyle10K and PERSONALITY-CAPTIONS datasets, outperforming prior methods in BLEU, CIDEr, ROUGE-L, and SPICE metrics while maintaining strong style alignment and contextual relevance.

ABSTRACT

In this paper, we build a multi-style generative model for stylish image captioning which uses multi-modality image features, ResNeXt features and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decode them to captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITY-CAPTIONS dataset and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc. A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions.

Motivation & Objective

• To address the limitations of existing multi-style image captioning models that struggle with contextual grounding and style consistency.
• To improve caption quality by integrating multi-modality features—ResNeXt visual features and text features from DenseCap—into a unified captioning framework.
• To enable a single model to generate diverse, stylistically accurate captions by fusing local (region-based) and global (image-level) visual features with style conditioning.
• To evaluate the effectiveness of multi-modality fusion and multi-style components through quantitative and qualitative analysis on benchmark datasets.
• To provide insights into caption generation failures by analyzing model behavior under incomplete or erroneous input features.

Proposed method

• The 3M model employs a Multi-UPDOWN architecture with two parallel encoder-decoder blocks, each processing a distinct modality: ResNeXt features and DenseCap text features.
• Each modality is encoded using a dedicated UPDOWN attention mechanism that selects relevant visual and textual features at each decoding step.
• A style vector, derived from a one-hot encoded personality label via learned embedding and linear projection, is concatenated with word embeddings to condition the generation process.
• Multi-modality features are fused through late interaction in the decoder, where the model attends to both visual and textual features while conditioning on the style vector.
• The model uses soft fusion of features from two pre-trained networks: ResNeXt for global image representation and DenseCap for dense, localized captioning features.
• Training is performed using supervised learning with cross-entropy loss, and the model is evaluated using standard NLP metrics including BLEU, ROUGE-L, CIDEr, SPICE, and METEOR.

Experimental results

Research questions

• RQ1 Can multi-modality features (ResNeXt and DenseCap) improve the quality and diversity of multi-style image captions compared to single-modality baselines?
• RQ2 Does the integration of a multi-style component via style vectors enhance the model’s ability to generate captions aligned with specific personalities?
• RQ3 How does the fusion of local (dense caption) and global (ResNeXt) features affect caption relevance and contextual accuracy?
• RQ4 To what extent does the model maintain style consistency and image context when input text features are noisy or incorrect?
• RQ5 What is the contribution of each architectural component (multi-style, multi-modality, Multi-UPDOWN) to overall caption quality, as shown by ablation studies?

Key findings

• The 3M model outperforms the baseline UPDOWN model across all NLP metrics on both FlickrStyle10K and PERSONALITY-CAPTIONS datasets, demonstrating the effectiveness of multi-modality fusion.
• On the FlickrStyle10K dataset, 3M achieves higher CIDEr (26.8), ROUGE-L (50.2), and SPICE (24.1) scores than the state-of-the-art SF-LSTM model, despite SF-LSTM being designed for single-style generation.
• The ablation study confirms that removing the multi-style component leads to a significant drop in all metrics, proving its critical role in style control and caption diversity.
• The model with only text features generates fewer unique words and lower expressiveness, while the model with only ResNeXt features struggles with relevance; the full model achieves the best balance in ROUGE-L (50.2), CIDEr (26.8), and SPICE (24.1).
• Qualitative analysis shows that when valid text features are available, the model leverages them effectively (e.g., R2-R3 in Figure 3), but when they are faulty, it relies on ResNeXt features to maintain coherence (e.g., R1 in Figure 3).
• The model can generate multiple captions with different styles from the same image, and it can identify and contextualize errors—such as incomplete captions or incorrect phrases—by analyzing feature inputs and style conditioning.