[Paper Review] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Rinon Gal, Yuval Alaluf+5 more|arXiv (Cornell University)|Aug 2, 2022

Multimodal Machine Learning ApplicationsComputer Science464 citations

TL;DR

The paper introduces Textual Inversion, learning a new pseudo-word in a frozen text-to-image model’s embedding space from 3–5 images, enabling personalized, language-guided generation and composition of user-provided concepts without fine-tuning the model.

ABSTRACT

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io

Motivation & Objective

• Enable personalized text-to-image generation for user-provided concepts using a frozen model.
• Represent new concepts with a single learned embedding (pseudo-word) in the text encoder.
• Preserve model priors while enabling fine-grained visual reconstruction and editing.
• Explore the embedding space with GAN-inversion inspired extensions and evaluate fidelity across tasks.
• Show applications from object variation, style transfer, composition, and bias reduction.

Proposed method

• Operate on latent diffusion models (LDMs) with a fixed text encoder and diffusion denoiser.
• Represent a new concept S* as a learnable embedding v* replacing the embedding of a placeholder token in the text encoder.
• Optimize v* using a reconstruction objective that minimizes the LDM loss over a small set of example images labeled with prompts like 'A photo of S*'.
• Keep cθ (the text encoder) and εθ (the denoiser) fixed during optimization of v*.
• Use neutral context prompts drawn from CLIP-ImageNet templates to guide generation during optimization.
• Experiment with hyperparameters close to LDM defaults (e.g., 5,000 optimization steps) and report qualitative and quantitative analyses.

Experimental results

Research questions

• RQ1 Can a single new embedding (pseudo-word) capture a user-provided concept with high fidelity across varied contexts?
• RQ2 How does embedding-based personalization compare to caption-based or image-prompt baselines in fidelity and editability?
• RQ3 What roles do extended embedding spaces, progressive or per-image tokens, and regularization play in reconstruction quality and generalization?
• RQ4 Can learned pseudo-words be used for style transfer, concept composition, or bias reduction without retraining the base model?
• RQ5 What are the limitations of composing multiple pseudo-words in complex relational prompts?

Key findings

• A single pseudo-word embedding can capture unique concepts and support diverse compositions in prompts.
• Pseudowords enable higher visual fidelity to the target concept than caption-based or discriminative baselines.
• Extensions inspired by GAN inversion (multi-word, progressive updates, regularization) offer limited or no clear gains over the base textual inversion.
• Embeddings can be used for style transfer, composition of multiple concepts, and bias reduction with small curated datasets.
• Learned pseudo-words can be integrated with downstream models like Blended Latent Diffusion for localized editing.
• The approach facilitates image variations, text-guided synthesis, and abstract concept representations (e.g., styles).