[Paper Review] GlyphDiffusion: Text Generation as Image Generation

Junyi Li, Wayne Xin Zhao+2 more|arXiv (Cornell University)|Apr 25, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

GlyphDiffusion proposes a novel text generation framework that treats conditional text generation as a text-guided image generation task by rendering target text as high-fidelity glyph images. By leveraging continuous diffusion models on these visual representations and incorporating a text grounding module, the method achieves state-of-the-art performance on conditional text generation tasks, outperforming both autoregressive and non-autoregressive models as well as prior diffusion-based approaches.

ABSTRACT

Diffusion models have become a new generative paradigm for text generation. Considering the discrete categorical nature of text, in this paper, we propose GlyphDiffusion, a novel diffusion approach for text generation via text-guided image generation. Our key idea is to render the target text as a glyph image containing visual language content. In this way, conditional text generation can be cast as a glyph image generation task, and it is then natural to apply continuous diffusion models to discrete texts. Specially, we utilize a cascaded architecture (ie a base and a super-resolution diffusion model) to generate high-fidelity glyph images, conditioned on the input text. Furthermore, we design a text grounding module to transform and refine the visual language content from generated glyph images into the final texts. In experiments over four conditional text generation tasks and two classes of metrics (ie quality and diversity), GlyphDiffusion can achieve comparable or even better results than several baselines, including pretrained language models. Our model also makes significant improvements compared to the recent diffusion model.

Motivation & Objective

• To address the challenge of adapting continuous diffusion models to discrete text generation, which is hindered by the inherent categorical nature of tokens.
• To overcome instability in training diffusion models for text by using fixed, visual targets (glyph images) instead of learning dynamic continuous representations.
• To improve text generation quality and diversity by leveraging the strong image generation capabilities of cascaded diffusion models on rendered text images.
• To bridge the gap between vision-based diffusion models and natural language generation through a novel image-to-text refinement pipeline.

Proposed method

• Rendering target text as glyph images to convert discrete text generation into a continuous image generation task.
• Employing a cascaded diffusion architecture with a base model and a super-resolution model to generate high-fidelity glyph images.
• Using classifier-free guidance conditioned on frozen T5-encoded text embeddings to improve content fidelity in generated glyph images.
• Introducing a text grounding module that refines visual language content from generated glyph images into coherent, final text outputs.
• Training the diffusion model on fixed glyph image targets to stabilize the denoising loss and avoid distribution shift during training.
• Utilizing a frozen T5 model to encode input semantics, ensuring consistent and meaningful conditioning for image generation.

Experimental results

Research questions

• RQ1 Can text generation be effectively reformulated as a conditional image generation task using rendered glyph images?
• RQ2 Does using fixed visual targets (glyph images) stabilize the training of diffusion models for discrete sequences compared to learning continuous representations?
• RQ3 Can cascaded diffusion models with classifier-free guidance generate high-fidelity glyph images that preserve semantic and syntactic accuracy?
• RQ4 To what extent does the text grounding module improve the quality and fluency of generated text compared to direct image-to-text decoding?
• RQ5 How does GlyphDiffusion compare to autoregressive, non-autoregressive, and prior diffusion-based text generation models in terms of quality and diversity?

Key findings

• GlyphDiffusion achieves over 50% improvement in BLEU and ROUGE-L scores compared to autoregressive and non-autoregressive baselines on conditional text generation tasks.
• On the Quasar-T dataset, GlyphDiffusion improves BLEU by +2.54 points over the previous state-of-the-art diffusion model.
• On the GYAFC dataset, it achieves a +2.24 improvement in Diverse-4, indicating superior generation diversity.
• The model outperforms both strong autoregressive and non-autoregressive models across all four evaluated conditional text generation tasks.
• The ablation study shows that removing the text grounding module leads to a significant drop in generation quality, confirming its critical role.
• The use of fixed glyph image targets prevents denoising loss collapse, enabling stable training of continuous diffusion models on discrete text.