[Paper Review] Design Guidelines for Prompt Engineering Text-to-Image Generative Models

Vivian Liu, Lydia B. Chilton|arXiv (Cornell University)|Sep 14, 2021

Aesthetic Perception and AnalysisNeuroscience32 citations

TL;DR

The paper analyzes how prompt wording, random seeds, iteration length, and style/subject choices affect text-to-image generations (over 5 experiments and 5493 generations) and derives practical design guidelines for better outcomes.

ABSTRACT

Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concrete subjects as well as 51 abstract and figurative styles. From this evaluation, we present design guidelines that can help people produce better outcomes from text-to-image generative models.

Motivation & Objective

• Investigate how prompt keywords and model hyperparameters influence the quality and coherence of text-to-image generations.
• Systematically evaluate prompts structured as 'SUBJECT in the style of STYLE' across many subjects and styles.
• Identify success and failure modes and translate findings into actionable design guidelines for end users.

Proposed method

• Use VQGAN+CLIP with 256x256 images, 300 optimization steps per image to generate prompts across 51 subjects and 12 styles in Experiment 1.
• Test nine permutations of prompt formulations for each subject-style pair to assess the impact of prompt wording.
• Perturb seeds (random initializations) and analyze whether different seeds produce significantly different generations.
• Vary the length of optimization (iterations) to determine how it correlates with perceived quality.
• Test 51 styles across 12 subjects to assess breadth of stylistic representations and potential biases.
• Annotate generations with human raters and perform statistical tests (Fisher’s exact test, Chi-square, Cohen’s kappa) to determine significance.

Figure 1. An example grid of text-to-image generations generated from the following prompt template: ”SUBJECT in the style of STYLE”. We analyze over 5000 generations in a series of five experiments involving 51 subjects and 51 styles to study what prompt parameters and hyperparameters can help peop

Experimental results

Research questions

• RQ1 Do different phrasings of the same keywords yield significantly different generations?
• RQ2 Does the random seed significantly affect generation quality for a fixed prompt?
• RQ3 How does the length of optimization influence generation quality and user preference?
• RQ4 How well can a wide breadth of styles be represented by the model, and are there style biases?
• RQ5 How do subject and style interact to affect generation outcomes?

Key findings

• Prompt permutations: no significant difference across nine prompt variants; focus on subject/style keywords rather than connecting words.
• Seed variation: seed choice significantly affects generation quality; recommend generating 3–9 seeds per prompt to capture variability.
• Length of optimization: shorter runs (100–500 iterations) are often preferred; 300 iterations suggested as a good default.
• Breadth of styles: model performance varies across 51 styles, with identifiable success modes in color, technique, spatial relationships, and motifs; observed style-specific biases.
• Overall, style and subject interact with the model’s capabilities, enabling qualitative success modes but with variance across styles.

Figure 2. For Experiment 1, annotators judged 3x3 grids where generations from different prompt permutations were arranged randomly. Annotators evaluated 143 grids of generations for significantly better generations as well significantly worse generations (outliers in generation quality). We found n