[Paper Review] PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

Yingchaojie Feng, Xingbo Wang+6 more|arXiv (Cornell University)|Jul 18, 2023

Image Retrieval and Classification Techniques74 references6 citations

TL;DR

PromptMagician is a visual analytics system that supports interactive prompt engineering for text-to-image generation by retrieving similar image-prompt pairs from DiffusionDB and recommending semantically relevant keywords via a CLIP-based, hierarchical clustering approach. It enables users to explore, evaluate, and refine prompts through multi-level visualizations, significantly improving prompt effectiveness and creativity support in text-to-image generation.

ABSTRACT

Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.

Motivation & Objective

• Address the challenge of ineffective prompt creation in text-to-image generation due to natural language ambiguity and complexity.
• Support novice and expert users in iteratively refining prompts and hyper-parameters through interactive visualization and recommendation.
• Provide personalized exploration of image results and prompt keywords using cross-modal embedding and user-defined evaluation criteria.
• Facilitate improved prompt engineering by identifying and recommending high-impact, relevant keywords from a large-scale prompt-image corpus (DiffusionDB).
• Enable deeper user understanding of prompt-image relationships through co-visualization of keywords and their corresponding image clusters.

Proposed method

• Leverages CLIP embeddings to encode retrieved images from DiffusionDB and group them into hierarchical clusters based on semantic similarity.
• Applies cluster-level TF-IDF to identify special (important and relevant) prompt keywords from the prompts associated with each image cluster.
• Matches recommended keywords to their most relevant image clusters and visualizes them alongside the clusters in a 2D embedding space for interactive exploration.
• Integrates a multi-view interface with four components: Model Input View, Image Browser View, Image Evaluation View, and Local Exploration View for guided refinement.
• Supports user-defined evaluation criteria (e.g., 'beautiful', 'good') to filter and focus on image subsets of interest.
• Employs a prompt recommendation model trained on design guidelines that prioritize keywords over sentence structures to improve prompt quality.

Figure 1 : The \name framework consists of four major components. It enables users to (A) specify model input for text-to-image creation. \name (B) generates a set of images using Stable Diffusion and (C) identifies related prompt keywords for recommendations. (D) Both the image results and prompt k

Experimental results

Research questions

• RQ1 How can a visual analytics system effectively support users in exploring and refining text-to-image prompts through interactive visualization and recommendation?
• RQ2 To what extent can a CLIP-based, hierarchical keyword recommendation model improve prompt engineering for text-to-image generation?
• RQ3 How do users perceive the usability and effectiveness of the system in discovering relevant prompt keywords and achieving desired image outputs?
• RQ4 Can multi-criteria image evaluation and co-visualization of keywords and images enhance user understanding of prompt effects on generation results?
• RQ5 How does the system support iterative refinement of prompts and hyper-parameters in real-world creative workflows?

Key findings

• The system significantly improves prompt engineering by recommending semantically relevant and high-impact keywords derived from a large-scale prompt-image corpus.
• User studies and expert interviews confirm that PromptMagician enhances creativity support and enables users to discover effective prompt keywords more efficiently.
• The multi-level visualization interface allows users to explore image clusters and their associated keywords, improving understanding of prompt effects on image generation.
• The integration of user-defined evaluation criteria (e.g., 'beautiful', 'detailed') enables focused exploration and filtering of image subsets, increasing usability.
• The system demonstrates strong usability and effectiveness in two real-world usage scenarios, supporting iterative refinement and personalized exploration.
• The prompt recommendation model based on hierarchical clustering and TF-IDF achieves high relevance in identifying keywords that align with user intent and image content.

Figure 2 : The pipeline of the prompt recommendation model involves five steps: (A) retrieving similar images from the DiffusionDB dataset; (B) embedding them according to their semantics; (C) arranging them into hierarchical clusters; (D) mining special and related keywords from the prompts in the