[论文解读] Highly Personalized Text Embedding for Image Manipulation by Stable Diffusion

Inhwa Han, Serin Yang+2 more|arXiv (Cornell University)|Mar 15, 2023

Generative Adversarial Networks and Image Synthesis被引用 10

一句话总结

本文提出 HiPer，一种简单的、无需微调的方法，通过将 CLIP 嵌入分解为高度个性化的尾部和语义头部来个性化 Stable Diffusion，从而实现对背景、纹理和运动的单图像、文本引导的操作。

ABSTRACT

Diffusion models have shown superior performance in image generation and manipulation, but the inherent stochasticity presents challenges in preserving and manipulating image content and identity. While previous approaches like DreamBooth and Textual Inversion have proposed model or latent representation personalization to maintain the content, their reliance on multiple reference images and complex training limits their practicality. In this paper, we present a simple yet highly effective approach to personalization using highly personalized (HiPer) text embedding by decomposing the CLIP embedding space for personalization and content manipulation. Our method does not require model fine-tuning or identifiers, yet still enables manipulation of background, texture, and motion with just a single image and target text. Through experiments on diverse target texts, we demonstrate that our approach produces highly personalized and complex semantic image edits across a wide range of tasks. We believe that the novel understanding of the text embedding space presented in this work has the potential to inspire further research across various tasks.

研究动机与目标

• 证明高度个性化的文本嵌入在实现由文本提示引导的语义编辑的同时，能够保留主体身份。
• 消除扩散模型驱动的图像操控中对模型微调或标识符的个性化需求。
• 展示仅用单一源图像和目标文本即可实现对背景、纹理和运动的操控。

提出的方法

• 将源 CLIP 嵌入分解为语义部分和尾部部分，将尾部保持为 HiPer 嵌入以保留身份信息。
• 仅优化 HiPer 尾部嵌入（N 个令牌），同时保持语义头固定，利用扩散模型损失。
• 在推理阶段通过将目标语义嵌入与 HiPer 尾部拼接来构建复合嵌入，以驱动图像生成。
• 在 Stable Diffusion 的潜在空间中执行优化，而不是在图像空间。
• 使用简短的训练过程（大约 3 分钟，1000 次优化步骤，N=5），无需模型微调或标识符。

Figure 1 : Image manipulation results with highly personalized (HiPer) text embeddings. In the upper row, the identities of the rabbit and the dog are well preserved while adequately manipulating the images to align with target texts. In the bottom row, not only motion and background, but also textu

实验结果

研究问题

• RQ1 一个小型、高度个性化的尾部嵌入是否能够在实现灵活的目标文本驱动编辑的同时保留主体身份？
• RQ2 将 CLIP 嵌入分解为个性化尾部和语义头部是否在不微调扩散模型的情况下提升对多方面编辑（运动、背景、纹理）的效果？
• RQ3 个性化令牌数量 N 对身份保持和编辑保真度有何影响？

主要发现

• HiPer 在保持源身份的同时实现跨越运动、背景和纹理的高度个性化编辑。
• 仅对 HiPer 尾部（N=5）在单一源图像下的优化，在 Stable Diffusion 下的结果与 DreamBooth、Textual Inversion 和 Imagic 基线相比具有竞争力甚至优越。
• 增大 N 会导致过拟合和源图像的身份信息泄露，而过小的 N 会降低身份保持。
• 跨注意力分析显示个性化嵌入与标准嵌入触发的注意力图不同，支持个性化与操控的分离。
• 该方法无需模型微调，训练时间约为三分钟。

Figure 2 : The proposed method. (Training) First, the source text prompt, which have the meaning of source image, is converted to text embedding. Some parts of text embedding, which have no information, are removed. The informative target embedding part and the personalized embedding is concatenated