[Paper Review] Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning

Guisheng Liu, Yi Li+4 more|arXiv (Cornell University)|Sep 10, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

Prefix-diffusion proposes a lightweight, diffusion-based image captioning model that injects continuous prefix image embeddings into the denoising process to generate diverse, fluent, and relevant captions. By leveraging a frozen CLIP encoder and a small trainable mapping network, it reduces parameters by over 38% compared to CLIP-based baselines while achieving state-of-the-art performance on newer metrics like Dist-3 and vocabulary usage.

ABSTRACT

While impressive performance has been achieved in image captioning, the limited diversity of the generated captions and the large parameter scale remain major barriers to the real-word application of these systems. In this work, we propose a lightweight image captioning network in combination with continuous diffusion, called Prefix-diffusion. To achieve diversity, we design an efficient method that injects prefix image embeddings into the denoising process of the diffusion model. In order to reduce trainable parameters, we employ a pre-trained model to extract image features and further design an extra mapping network. Prefix-diffusion is able to generate diverse captions with relatively less parameters, while maintaining the fluency and relevance of the captions benefiting from the generative capabilities of the diffusion model. Our work paves the way for scaling up diffusion models for image captioning, and achieves promising performance compared with recent approaches.

Motivation & Objective

• To address the limited diversity in autoregressive image captioning models that generate repetitive or generic captions.
• To reduce the large parameter count typical of multimodal captioning models, improving real-world deployability.
• To enable parallel, non-autoregressive caption generation that avoids error accumulation from left-to-right autoregressive decoding.
• To integrate continuous diffusion models effectively into image captioning by bridging vision-language modality gaps.
• To achieve strong generalization across domains, including zero-shot cross-dataset evaluation.

Proposed method

• Injects learned prefix image embeddings into the denoising process of a continuous diffusion model to condition text generation on visual content.
• Uses a pre-trained CLIP image encoder as a frozen feature extractor to reduce trainable parameters.
• Employs a lightweight, trainable mapping network to project CLIP image features into the text space for diffusion modeling.
• Applies a truncated linear noise schedule in the forward diffusion process to preserve semantic integrity and improve generation quality.
• Generates diverse captions by sampling from multiple noise seeds during inference, avoiding beam search limitations.
• Selects top candidate captions using CLIP-based image-text similarity, balancing relevance and diversity.

Experimental results

Research questions

• RQ1 Can continuous diffusion models be effectively adapted for diverse image captioning while maintaining fluency and relevance?
• RQ2 How can parameter efficiency be improved in vision-language captioning without sacrificing performance?
• RQ3 Does injecting visual prefixes into the diffusion denoising process enhance caption diversity and quality?
• RQ4 How does the model generalize across domains, particularly in zero-shot cross-dataset evaluation?
• RQ5 What impact do different noise schedules have on caption quality and semantic preservation?

Key findings

• Prefix-diffusion achieves a Dist-3 score of 109.3 and a vocabulary usage score of 107.8, representing a 6.3 and 3.1 improvement over baselines, respectively.
• The model reduces trainable parameters by over 38% compared to existing CLIP-based methods while maintaining or improving performance on CIDEr, BLEU, and ROUGE-L metrics.
• On cross-domain evaluation, Prefix-diffusion outperforms all baselines on COCO → Flickr30k and shows strong generalization, especially when pre-trained on larger datasets.
• The truncated linear noise schedule yields superior results, improving caption quality and descriptive accuracy compared to cosine or standard schedules.
• Ablation studies confirm that selecting top-k captions using CLIP similarity boosts CIDEr from 105.2 to 109.3, though excessive candidates may reduce fluency.
• The model generates diverse captions with rich expression and varied phrasing, aligning better with human-generated captions than autoregressive models.