[Paper Review] Conditional Text Image Generation with Diffusion Models

Yuanzhi Zhu, Zhaohai Li+3 more|arXiv (Cornell University)|Jun 19, 2023

Handwritten Text Recognition TechniquesComputer Science3 citations

TL;DR

This paper proposes CTIG-DM, a diffusion model-based method for conditional text image generation that uses image, text, and style conditions to generate high-fidelity, diverse text images. It achieves state-of-the-art performance in text recognition accuracy, OOV word generation, and domain adaptation by enabling four distinct generation modes.

ABSTRACT

Current text recognition systems, including those for handwritten scripts and scene text, have relied heavily on image synthesis and augmentation, since it is difficult to realize real-world complexity and diversity through collecting and annotating enough real text images. In this paper, we explore the problem of text image generation, by taking advantage of the powerful abilities of Diffusion Models in generating photo-realistic and diverse image samples with given conditions, and propose a method called Conditional Text Image Generation with Diffusion Models (CTIG-DM for short). To conform to the characteristics of text images, we devise three conditions: image condition, text condition, and style condition, which can be used to control the attributes, contents, and styles of the samples in the image generation process. Specifically, four text image generation modes, namely: (1) synthesis mode, (2) augmentation mode, (3) recovery mode, and (4) imitation mode, can be derived by combining and configuring these three conditions. Extensive experiments on both handwritten and scene text demonstrate that the proposed CTIG-DM is able to produce image samples that simulate real-world complexity and diversity, and thus can boost the performance of existing text recognizers. Besides, CTIG-DM shows its appealing potential in domain adaptation and generating images containing Out-Of-Vocabulary (OOV) words.

Motivation & Objective

• To address the challenge of limited real-world text image data in training robust text recognition models.
• To develop a conditional text image generation framework that ensures content validity, image fidelity, and diversity.
• To explore the potential of diffusion models in generating complex text images, including OOV and rare characters.
• To improve text recognition performance through data synthesis and domain adaptation using generated samples.

Proposed method

• Introduces a conditional diffusion model with three key conditions: image condition, text condition, and style condition for controlling content, attributes, and writing style.
• Employs a conditional encoder to generate the three conditions from input text, style embeddings, and optional image priors.
• Utilizes a denoising diffusion process conditioned on the three inputs to generate photo-realistic text images from random noise.
• Supports four generation modes—synthesis, augmentation, recovery, and imitation—by combining or configuring the three conditions.
• Leverages CLIP-based pre-training for cross-modal alignment between text and image conditions in the diffusion process.
• Applies radical sequence decomposition for complex script generation, such as OOV Chinese characters and ancient scripts.

Experimental results

Research questions

• RQ1 Can diffusion models effectively generate high-fidelity, diverse, and content-valid text images for data augmentation in text recognition?
• RQ2 How do the three proposed conditions (text, style, image) contribute to controlling image quality and semantic consistency in text image generation?
• RQ3 To what extent can CTIG-DM improve text recognition performance through synthetic data in low-resource or out-of-domain settings?
• RQ4 Can CTIG-DM generate realistic images of OOV or rare characters, such as non-existing Chinese radicals or ancient scripts?
• RQ5 How does CTIG-DM compare to existing GAN-based and autoregressive methods in generating diverse and valid text images?

Key findings

• CTIG-DM achieves a FID of 25.52 on OOV image generation for the IAM dataset, significantly outperforming prior methods like CG-GAN (104.81) and SLOGAN (97.81).
• On the CVL dataset, CTIG-DM reduces WER by 16.40% and CER by 7.60% compared to the baseline, demonstrating strong domain adaptation capability.
• Generated images from CTIG-DM are nearly indistinguishable from real images, as confirmed by human evaluation in both handwritten and scene text settings.
• The method successfully generates non-existent Chinese characters and ancient rare characters by combining radical sequences and style priors.
• The four generation modes—synthesis, augmentation, recovery, and imitation—enable flexible application across diverse text recognition scenarios.
• The model shows strong generalization to unseen writers and rare character distributions, indicating robustness and scalability.