[Paper Review] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Nataniel Ruiz, Yuanzhen Li+4 more|arXiv (Cornell University)|Aug 25, 2022

Generative Adversarial Networks and Image SynthesisComputer Science108 citations

TL;DR

DreamBooth fine-tunes a pretrained text-to-image diffusion model using a few subject images to bind a unique identifier to that subject, enabling novel, context-rich renders while preserving subject identity.

ABSTRACT

Large text-to-image models achieved a remarkable leap in the evolution of AI, enabling high-quality and diverse synthesis of images from a given text prompt. However, these models lack the ability to mimic the appearance of subjects in a given reference set and synthesize novel renditions of them in different contexts. In this work, we present a new approach for "personalization" of text-to-image diffusion models. Given as input just a few images of a subject, we fine-tune a pretrained text-to-image model such that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel photorealistic images of the subject contextualized in different scenes. By leveraging the semantic prior embedded in the model with a new autogenous class-specific prior preservation loss, our technique enables synthesizing the subject in diverse scenes, poses, views and lighting conditions that do not appear in the reference images. We apply our technique to several previously-unassailable tasks, including subject recontextualization, text-guided view synthesis, and artistic rendering, all while preserving the subject's key features. We also provide a new dataset and evaluation protocol for this new task of subject-driven generation. Project page: https://dreambooth.github.io/

Motivation & Objective

• Address how to personalize text-to-image diffusion models to imitate a specific subject from a few images.
• Bind a unique identifier to a subject so it can be synthesized in novel contexts guided by text prompts.
• Mitigate language drift and loss of diversity during fine-tuning with a class-specific prior preservation loss.
• Demonstrate subject recontextualization, view synthesis, and artistic rendering while preserving subject fidelity.
• Provide a dataset and evaluation protocol for subject-driven generation.

Proposed method

• Fine-tune a pretrained diffusion model using 3-5 subject images paired with prompts containing a unique identifier and the class noun (e.g., “A [V] dog”).
• Represent the subject with rare token identifiers to minimize prior from common language and facilitate binding to the subject.
• Use a class-specific prior preservation loss that supervises the model with its own generated samples to prevent language drift and preserve prior diversity.
• Train all layers of the model to maximize fidelity and enable diverse outputs, with modest iteration counts (≈1000) and small datasets.
• Evaluate subject fidelity with DINO and CLIP-I, and prompt fidelity with CLIP-T, plus a user study comparing against baselines.
• Provide a dataset of 30 subjects (objects and live subjects) and 25 prompts, with four images per subject/prompt (≈3,000 images total).

Figure 2 : Subject-driven generation. Given a particular clock (left), it is hard to generate it while maintaining high fidelity to its key visual features (second and third columns showing DALL-E2 [ 54 ] image-guided generation and Imagen [ 61 ] text-guided generation; text prompt used for Imagen:

Experimental results

Research questions

• RQ1 Can a diffusion model be personalized to a specific subject from a handful of images without losing its prior knowledge?
• RQ2 Does binding a rare-identifier token to a subject enable faithful and diverse generation in new contexts?
• RQ3 How effective is a class-specific prior preservation loss at preventing language drift and maintaining prior diversity during fine-tuning?
• RQ4 What are the quantitative and human-evaluated gains of DreamBooth over existing baselines in subject fidelity and prompt fidelity?
• RQ5 What datasets and evaluation protocols realistically measure subject-driven generation performance?

Key findings

• DreamBooth with Imagen achieves higher subject and prompt fidelity than DreamBooth with Stable Diffusion and outperforms Textual Inversion in both metrics.
• A user study shows DreamBooth is preferred over Textual Inversion for subject fidelity and prompt fidelity across multiple prompts.
• The autogenous prior preservation loss helps counter language drift and maintain prior diversity, enabling more varied subject renders.
• Training with the proposed loss allows effective 3-5 image personalization within roughly 5 minutes on appropriate hardware.
• DreamBooth enables recontextualization, novel view synthesis, and property modifications while preserving the subject’s identity.
• A new dataset and evaluation protocol for subject-driven generation are provided to support future research.

Figure 3 : Fine-tuning. Given $\sim 3-5$ images of a subject we fine-tune a text-to-image diffusion model with the input images paired with a text prompt containing a unique identifier and the name of the class the subject belongs to (e.g., “A [V] dog”), in parallel, we apply a class-specific prior