[论文解读] Improving Diffusion Models for Scene Text Editing with Dual Encoders

Jiabao Ji, Guanhua Zhang+5 more|arXiv (Cornell University)|Apr 12, 2023

Generative Adversarial Networks and Image Synthesis被引用 6

一句话总结

该论文提出 DiffSte，一种双编码器扩散模型，通过整合字符编码器以提升拼写准确性，并引入指令编码器以实现风格控制。该模型通过指令微调进行训练，实现了对未见过的字体、混合字体风格以及自然语言风格描述的零样本泛化，在五个数据集上均优于现有方法，在文本正确性、图像自然度和风格可控性方面表现更优。

ABSTRACT

Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this task rely on style-transfer models that crop out text regions and feed them into image transfer models, such as GANs. However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusion models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design, which includes a character encoder for better text legibility and an instruction encoder for better style control. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. We evaluate our approach on five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability. Our code is publicly available. https://github.com/UCSB-NLP-Chang/DiffSTE

研究动机与目标

• 解决现有扩散模型在场景文本编辑中生成准确、有风格文本的局限性。
• 克服如 Stable Diffusion 等最先进扩散模型中存在文本可读性差和风格控制弱的问题。
• 实现对未见过的字体变体（如斜体、粗体）及混合字体风格的零样本泛化。
• 支持灵活的自然语言指令进行风格控制，例如“呈绿色”或“如烈焰般燃烧”。
• 在任意形状的掩码内实现文本渲染，同时保持布局和方向的一致性。

提出的方法

• 提出双编码器架构：字符编码器以字符级别处理目标文本，指令编码器则源自 Stable Diffusion 的原始文本编码器。
• 在图像生成过程中，利用交叉注意力机制同时关注字符级嵌入和指令嵌入。
• 聚合字符嵌入与指令嵌入，引导扩散模型生成拼写正确且具有指定风格的文本。
• 采用指令微调框架，在配对的指令-图像数据上微调模型，使其学会将自然语言描述映射到视觉文本外观。
• 以预训练的扩散模型（如 Stable Diffusion）为基础，通过最小的架构修改进行适配。
• 通过在掩码区域和文本指令上联合条件控制，支持任意形状的掩码，实现布局感知的文本放置。

Figure 1: Text editing results by GAN-based Mostel , diffusion model SD , and our method DiffSte . A reference Rendered image with desired content and style is fed to Mostel as style guidance. SD and our method are prompted by instruction Write a grass-colored word “DiffSTE” in BadScript font .

实验结果

研究问题

• RQ1 双编码器设计能否提升基于扩散模型的场景文本编辑中的文本可读性和拼写准确性？
• RQ2 指令微调能否实现对未见过的字体风格（如斜体或粗体变体）的零样本泛化？
• RQ3 通过使用自然语言提示混合两种已知字体，模型能否泛化到新型字体组合？
• RQ4 模型能否准确理解多样化的自然语言描述（如“薰衣草色”或“烈焰般燃烧”）以实现精确的文本渲染？
• RQ5 模型能否在复杂非矩形掩码中生成具有正确布局和方向的文本？

主要发现

• DiffSte 在包括 COCOText、ICDAR13 和 TextOCR 在内的五个基准数据集上，均在文本正确性、图像自然度和风格可控性方面表现更优。
• 模型在未见过的字体风格（如斜体和粗体）上实现零样本泛化，且保真度高，如字体外推的定性结果所示。
• DiffSte 通过简单的自然语言提示（如“用字体 Font1 和 Font2 写”）成功生成混合字体。
• 模型能准确理解多样化的自然语言风格描述，如“芒果色”或“深海蓝”，展现出对语义差异的鲁棒性。
• DiffSte 即使在高度弯曲和不规则的掩码中，也能生成具有正确布局和方向的文本，如“the five”和“boxing”的示例所示。
• 指令微调框架使模型能够学习从自然语言指令到视觉文本外观的映射，而无需显式的风格标签。

Figure 3: Model structure of DiffSte with dual-encoder design. The input text instruction and the target text spelling are first processed by our instruction encoder $\textsc{Enc}_{\textrm{inst}}$ and character encoder $\textsc{Enc}_{\textrm{char}}$ , respectively. Both encodings are attended to the