[Paper Review] Diff-Font: Diffusion Model for Robust One-Shot Font Generation

Haibin He, Xinyuan Chen+5 more|arXiv (Cornell University)|Dec 12, 2022

Video Analysis and Summarization7 citations

TL;DR

Diff-Font proposes a diffusion model-based framework for robust one-shot font generation, treating font generation as a conditional generation task using content embeddings, style reference images, and fine-grained stroke/component conditions. It achieves state-of-the-art performance in style fidelity and structural integrity, especially for complex characters in Chinese and Korean scripts, with stable training and superior quantitative results on benchmark datasets.

ABSTRACT

Font generation is a difficult and time-consuming task, especially in those languages using ideograms that have complicated structures with a large number of characters, such as Chinese. To solve this problem, few-shot font generation and even one-shot font generation have attracted a lot of attention. However, most existing font generation methods may still suffer from (i) large cross-font gap challenge; (ii) subtle cross-font variation problem; and (iii) incorrect generation of complicated characters. In this paper, we propose a novel one-shot font generation method based on a diffusion model, named Diff-Font, which can be stably trained on large datasets. The proposed model aims to generate the entire font library by giving only one sample as the reference. Specifically, a large stroke-wise dataset is constructed, and a stroke-wise diffusion model is proposed to preserve the structure and the completion of each generated character. To our best knowledge, the proposed Diff-Font is the first work that developed diffusion models to handle the font generation task. The well-trained Diff-Font is not only robust to font gap and font variation, but also achieved promising performance on difficult character generation. Compared to previous font generation methods, our model reaches state-of-the-art performance both qualitatively and quantitatively.

Motivation & Objective

• To address the limitations of GAN-based one-shot font generation, including unstable training, poor style transfer fidelity, and failure on complex characters.
• To enable robust, high-fidelity font generation for glyph-rich languages such as Chinese and Korean, which require precise structural modeling.
• To develop a unified, conditional generation framework that decouples content, style, and structural components for improved control and quality.
• To demonstrate the effectiveness of diffusion models in font generation, a domain previously dominated by GANs.
• To achieve state-of-the-art performance in both qualitative and quantitative evaluations across multiple scripts and datasets.

Proposed method

• The method formulates font generation as a conditional diffusion process, where content is represented via learned embedding tokens, and style is conditioned on a single reference image.
• Fine-grained structural conditions—such as stroke or component maps—are incorporated to preserve character integrity, especially for complex glyphs in Chinese and Korean.
• A multi-conditioned denoising diffusion model is trained to generate images by iteratively denoising noise, guided by content, style, and structural conditions.
• The model is trained in two stages: first on a large-scale dataset with content and style conditions, and second fine-tuned with structural conditions for improved fidelity.
• Guidance scales for content and stroke conditions are optimized during inference to balance fidelity and diversity, with optimal settings found at s₁=3, s₂=3.
• The framework is language-agnostic and generalizes to Latin and Greek scripts by using only content and style conditions, demonstrating broad applicability.

Experimental results

Research questions

• RQ1 Can a diffusion model achieve stable and high-fidelity one-shot font generation for complex scripts like Chinese and Korean?
• RQ2 How does incorporating stroke and component-level structural conditions improve the generation of intricate characters?
• RQ3 Does the proposed conditional diffusion framework outperform existing GAN-based methods in both style transfer accuracy and structural preservation?
• RQ4 What is the optimal balance between content and structural guidance during inference for maximizing generation quality?
• RQ5 Can the framework generalize effectively to scripts with simpler structures, such as Latin and Greek, without additional structural conditioning?

Key findings

• Diff-Font achieves state-of-the-art performance on Chinese font generation, with a Fréchet Inception Distance (FID) of 16.20 on the test set, outperforming prior methods.
• On the Korean script benchmark, Diff-Font achieves an FID of 10.69, significantly lower than DG-Font (43.36) and MX-Font (47.05), demonstrating superior generation quality.
• The optimal guidance scales for content and stroke conditions were found to be s₁=3 and s₂=3, yielding the highest SSIM (0.722) and lowest RMSE (0.277) and LPIPS (0.104).
• Abllation studies show that using stroke count encoding improves generation quality over no stroke condition or one-bit encoding, reducing FID and LPIPS.
• The model generalizes effectively to Latin and Greek scripts, producing high-quality results without requiring structural annotations.
• Despite strong performance, failure cases persist for characters with extreme complexity or uncommon styles, indicating limitations in rare or out-of-distribution examples.