[Paper Review] CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning

Shitong Xu|arXiv (Cornell University)|Oct 10, 2022

Multimodal Machine Learning Applications7 citations

TL;DR

This paper proposes CLIP-Diffusion-LM, a non-autoregressive image captioning model that leverages a denoising diffusion language model for text generation, using CLIP features as conditioning. It achieves 0.2470 BLEU-4 on Flickr30k+Flickr8k with significantly fewer inference steps than autoregressive models, demonstrating the feasibility and efficiency of diffusion models in image captioning.

ABSTRACT

Image captioning task has been extensively researched by previous work. However, limited experiments focus on generating captions based on non-autoregressive text decoder. Inspired by the recent success of the denoising diffusion model on image synthesis tasks, we apply denoising diffusion probabilistic models to text generation in image captioning tasks. We show that our CLIP-Diffusion-LM is capable of generating image captions using significantly fewer inference steps than autoregressive models. On the Flickr8k dataset, the model achieves 0.1876 BLEU-4 score. By training on the combined Flickr8k and Flickr30k dataset, our model achieves 0.2470 BLEU-4 score. Our code is available at https://github.com/xu-shitong/diffusion-image-captioning.

Motivation & Objective

• To explore the application of diffusion models in non-autoregressive text generation for image captioning, addressing limitations of autoregressive models such as slow generation and inability to refine earlier tokens.
• To investigate the effectiveness of diffusion-based language modeling in generating high-quality image captions using pre-trained CLIP and DistilBERT components.
• To empirically evaluate design choices such as loss weighting, guidance scheduling, and feature fusion methods in the context of diffusion-based captioning.
• To provide insights into training tricks that improve caption quality and model convergence in diffusion-based image captioning.

Proposed method

• The model uses CLIP to extract image and text features, which are fused with the caption embedding via concatenation or element-wise addition.
• A diffusion language model based on DistilBERT is trained to denoise latent text sequences by predicting noise at each timestep, with the goal of reconstructing the original caption embedding.
• Classification-free guidance is applied during inference to improve generation quality without requiring a classifier.
• The loss function includes a denoising term $ L_{diffuseLM} $, a reconstruction term $ L_{recon} $, and a rounding term $ L_{round} $, with adaptive weighting to balance convergence and performance.
• The model is trained to predict $ x_0 $, the original caption embedding, rather than a later denoised state, which improves performance and convergence speed.
• Hyperparameter ablation studies are conducted on learning rate, loss weighting, and $ x_0 $-prediction vs. $ x_{s_t - n} $-prediction to identify optimal configurations.

Experimental results

Research questions

• RQ1 Can a diffusion-based language model effectively generate image captions in a non-autoregressive manner, achieving competitive performance with autoregressive models?
• RQ2 How do different loss function components, particularly the rounding term and adaptive weighting, affect model convergence and caption quality?
• RQ3 What is the impact of $ x_0 $-prediction versus $ x_{s_t - n} $-prediction on generation quality and training stability?
• RQ4 Which feature fusion method—concatenation or element-wise addition—yields better performance despite trade-offs in training stability?
• RQ5 How does the use of classification-free guidance influence the quality of generated captions in the absence of a separate classifier?

Key findings

• The model achieves a BLEU-4 score of 0.2470 on the combined Flickr8k and Flickr30k dataset, demonstrating strong performance with significantly fewer inference steps than autoregressive models.
• Using $ x_0 $-prediction instead of $ x_{s_t - n} $-prediction leads to better BLEU-4 scores (0.1549 vs. 0.1474) and lower $ L_{diffuseLM} $ loss (17.8 vs. 22.6), indicating improved convergence and performance.
• The concatenation-based feature fusion method outperforms element-wise addition, achieving 0.2337 BLEU-4 on the combined dataset despite higher training loss fluctuations.
• The addition of a rounding term $ L_{round} $ in the loss function is critical for model convergence, and adaptive weighting of this term improves performance.
• A learning rate of 5e-5 with linear decay schedule yields the best performance, with a BLEU-4 score of 0.2470 when training on the combined dataset.
• The model converges to a reasonable output in just 5 diffusion inference stages, highlighting its efficiency compared to autoregressive models that require steps proportional to sequence length.