[Paper Review] Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models

Lai Zeqiang, Zhu Xizhou+3 more|arXiv (Cornell University)|Oct 11, 2023

Topic Modeling5 citations

TL;DR

This paper introduces Mini-DALL·E 3, a prompting-based framework that enables interactive text-to-image (iT2I) generation by integrating large language models (LLMs) with off-the-shelf text-to-image diffusion models. By leveraging natural language conversations, users can iteratively refine images through multi-turn interactions, achieving high-quality, context-aware image generation without model fine-tuning, while preserving the LLM’s core capabilities in reasoning and question-answering.

ABSTRACT

The revolution of artificial intelligence content generation has been rapidly accelerated with the booming text-to-image (T2I) diffusion models. Within just two years of development, it was unprecedentedly of high-quality, diversity, and creativity that the state-of-the-art models could generate. However, a prevalent limitation persists in the effective communication with these popular T2I models, such as Stable Diffusion, using natural language descriptions. This typically makes an engaging image hard to obtain without expertise in prompt engineering with complex word compositions, magic tags, and annotations. Inspired by the recently released DALLE3 - a T2I model directly built-in ChatGPT that talks human language, we revisit the existing T2I systems endeavoring to align human intent and introduce a new task - interactive text to image (iT2I), where people can interact with LLM for interleaved high-quality image generation/edit/refinement and question answering with stronger images and text correspondences using natural language. In addressing the iT2I problem, we present a simple approach that augments LLMs for iT2I with prompting techniques and off-the-shelf T2I models. We evaluate our approach for iT2I in a variety of common-used scenarios under different LLMs, e.g., ChatGPT, LLAMA, Baichuan, and InternLM. We demonstrate that our approach could be a convenient and low-cost way to introduce the iT2I ability for any existing LLMs and any text-to-image models without any training while bringing little degradation on LLMs' inherent capabilities in, e.g., question answering and code generation. We hope this work could draw broader attention and provide inspiration for boosting user experience in human-machine interactions alongside the image quality of the next-generation T2I systems.

Motivation & Objective

• Address the challenge of poor user-friendliness in existing text-to-image models, which require expert prompt engineering for high-quality outputs.
• Overcome limitations in natural language understanding and prompt configuration complexity that hinder non-expert users from effectively generating desired images.
• Introduce a new task—interactive text-to-image (iT2I)—where users engage in multi-turn dialogues with LLM-powered agents to iteratively generate, edit, and refine images using natural language.
• Enable low-cost, training-free integration of iT2I capabilities into any existing LLM and text-to-image model combination.
• Preserve the inherent capabilities of LLMs—such as question-answering and code generation—while adding image generation and editing functionality.

Proposed method

• Use few-shot prompting and chain-of-thought prompting to guide LLMs in generating appropriate text prompts for diffusion-based T2I models.
• Integrate off-the-shelf T2I models (e.g., Stable Diffusion) with LLMs in a multi-turn dialogue setting to support image generation, editing, and refinement.
• Maintain conversation context to ensure consistency in multi-turn image editing, such as preserving character identity or scene elements.
• Design a prompting pipeline that maps natural language instructions (e.g., 'Make the dog bigger' or 'Don’t be a sticker') into structured prompts compatible with T2I models.
• Evaluate the framework across diverse LLMs—including GPT-3.5, GPT-4, Claude 3, LLAMA2, Baichuan2, Qwen, and InternLM—to assess generalization and robustness.
• Use a zero-shot evaluation setup with no model fine-tuning, relying solely on prompt engineering to enable iT2I functionality.

Figure 2 : The evolution of image generation systems.

Experimental results

Research questions

• RQ1 Can prompting techniques enable existing LLMs to support interactive, multi-turn text-to-image generation without fine-tuning?
• RQ2 How well do different LLMs—commercial and open-source—perform in generating contextually accurate and high-quality image prompts through natural language interaction?
• RQ3 To what extent does the proposed prompting-based approach preserve the LLM’s original capabilities in tasks like question-answering and code generation?
• RQ4 Can the system maintain visual consistency across multiple edits in a dialogue, such as preserving character identity or scene structure?
• RQ5 What are the limitations of open-source LLMs in handling iT2I tasks compared to commercial models?

Key findings

• Commercial LLMs such as GPT-3.5, GPT-4, and Claude 3 successfully generated appropriate image prompts and interleaved responses, demonstrating robust iT2I capability via prompting.
• Baichuan2-13B-Chat produced the best results among open-source LLMs, while Qwen and InternLM often refused to generate images despite prompting.
• ChatGLM2-6B-Chat generated images but failed to maintain correct text-image correspondence, indicating issues with prompt alignment.
• The prompting-based approach enabled zero-shot integration of iT2I functionality into any LLM and T2I model pair without retraining.
• The method preserved core LLM capabilities, including question-answering and code generation, with minimal degradation.
• Evaluation on MMLU showed that the prompting approach did not significantly impair LLM performance, with average scores of 59.34 (few-shot) and 56.37 (zero-shot), indicating strong retention of reasoning abilities.

Figure 3 : Illustrations of different human-machine interfaces for T2I systems.