[Paper Review] Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

Ling Yang, Zhaochen Yu+4 more|arXiv (Cornell University)|Jan 22, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes RPG (Recaption, Plan, and Generate), a training-free text-to-image diffusion framework that leverages multimodal LLMs for improved compositional generation. By using MLLMs to recaption prompts, plan region-wise image composition, and enable complementary regional diffusion, RPG outperforms SOTA models like DALL-E 3 and SDXL in complex prompt alignment and multi-object compositionality.

ABSTRACT

Diffusion models have exhibit exceptional performance in text-to-image generation and editing. However, existing methods often face challenges when handling complex text prompts that involve multiple objects with multiple attributes and relationships. In this paper, we propose a brand new training-free text-to-image generation/editing framework, namely Recaption, Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning ability of multimodal LLMs to enhance the compositionality of text-to-image diffusion models. Our approach employs the MLLM as a global planner to decompose the process of generating complex images into multiple simpler generation tasks within subregions. We propose complementary regional diffusion to enable region-wise compositional generation. Furthermore, we integrate text-guided image generation and editing within the proposed RPG in a closed-loop fashion, thereby enhancing generalization ability. Extensive experiments demonstrate our RPG outperforms state-of-the-art text-to-image diffusion models, including DALL-E 3 and SDXL, particularly in multi-category object composition and text-image semantic alignment. Notably, our RPG framework exhibits wide compatibility with various MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available at: https://github.com/YangLing0818/RPG-DiffusionMaster

Motivation & Objective

• Address the challenge of accurately generating complex, compositional images from intricate text prompts involving multiple objects, attributes, and relationships.
• Overcome limitations of existing layout-based and feedback-based methods that offer only coarse spatial control or require costly training and feedback collection.
• Enable training-free, controllable, and compositional image generation by leveraging the reasoning and planning capabilities of multimodal LLMs.
• Unify text-guided image generation and editing in a closed-loop system to improve semantic alignment and generalization across diverse prompts.
• Ensure wide compatibility with various MLLM architectures and diffusion backbones such as ControlNet and Stable Diffusion XL.

Proposed method

• Employ multimodal LLMs for multimodal recaptioning: transforming ambiguous or complex prompts into detailed, semantically rich subprompts to improve prompt comprehension.
• Introduce chain-of-thought planning via MLLMs to decompose the image generation task into subregions, assigning distinct subprompts to each region for structured composition.
• Design complementary regional diffusion: independently generate image content within non-overlapping subregions using region-specific subprompts, enhancing spatial precision and compositional control.
• Integrate a closed-loop feedback mechanism that uses MLLM-based image captioning to detect semantic discrepancies between generated and target images, enabling iterative refinement.
• Utilize MLLM-based reasoning to generate detailed rationales and precise instructions for region division and subprompt assignment, improving planning accuracy.
• Ensure compatibility with diverse diffusion backbones (e.g., ControlNet) and MLLM architectures (e.g., MiniGPT-4) by decoupling planning and generation stages.

Experimental results

Research questions

• RQ1 Can multimodal LLMs effectively decompose complex text prompts into semantically coherent, region-specific subprompts for improved image composition?
• RQ2 How does chain-of-thought reasoning in MLLMs enhance the planning of spatial layout and subregion assignment in text-to-image generation?
• RQ3 Can complementary regional diffusion outperform global diffusion in generating images with multiple overlapping or interacting objects and attributes?
• RQ4 To what extent does the closed-loop integration of recaptioning and feedback improve semantic alignment between text prompts and generated images?
• RQ5 How generalizable is the RPG framework across different MLLM architectures and diffusion model backbones without requiring fine-tuning?

Key findings

• RPG outperforms state-of-the-art models such as DALL-E 3 and SDXL in generating images that accurately reflect complex, compositional prompts, especially in multi-category object composition.
• The framework achieves superior text-image semantic alignment, as evidenced by improved performance on benchmark datasets measuring compositional accuracy and prompt adherence.
• Complementary regional diffusion enables precise, localized control over image generation, significantly reducing spatial misalignment and object overlap issues common in baseline diffusion models.
• The closed-loop recaptioning and feedback mechanism effectively identifies and corrects semantic discrepancies between generated images and input prompts, enhancing final image quality.
• RPG demonstrates strong compatibility with various MLLM architectures (e.g., MiniGPT-4) and diffusion backbones (e.g., ControlNet), enabling flexible deployment across different models.
• The method is training-free and operates entirely at inference time, avoiding the need for fine-tuning or reward model pretraining, which reduces computational and data overhead.