[Paper Review] Few-shot Font Generation by Learning Style Difference and Similarity

Xiao He, Mingrui Zhu+3 more|arXiv (Cornell University)|Jan 24, 2023

Video Analysis and Summarization4 citations

TL;DR

This paper proposes DS-Font, a few-shot font generation method that learns style differences between fonts and similarities within the same font using contrastive learning. By introducing a multi-layer style projector and a cluster-level contrastive style loss, it achieves superior style consistency and reduced artifacts compared to SOTA methods, outperforming them on key metrics like FID and style accuracy.

ABSTRACT

Few-shot font generation (FFG) aims to preserve the underlying global structure of the original character while generating target fonts by referring to a few samples. It has been applied to font library creation, a personalized signature, and other scenarios. Existing FFG methods explicitly disentangle content and style of reference glyphs universally or component-wisely. However, they ignore the difference between glyphs in different styles and the similarity of glyphs in the same style, which results in artifacts such as local distortions and style inconsistency. To address this issue, we propose a novel font generation approach by learning the Difference between different styles and the Similarity of the same style (DS-Font). We introduce contrastive learning to consider the positive and negative relationship between styles. Specifically, we propose a multi-layer style projector for style encoding and realize a distinctive style representation via our proposed Cluster-level Contrastive Style (CCS) loss. In addition, we design a multi-task patch discriminator, which comprehensively considers different areas of the image and ensures that each style can be distinguished independently. We conduct qualitative and quantitative evaluations comprehensively to demonstrate that our approach achieves significantly better results than state-of-the-art methods.

Motivation & Objective

• To address artifacts like local distortions and style inconsistency in few-shot font generation by modeling style relationships rather than relying on explicit disentanglement.
• To improve style representation by capturing both inter-style differences and intra-style similarities using contrastive learning.
• To develop a robust generator that maintains global character structure while transferring fine-grained style patterns from few reference samples.
• To design a multi-task patch discriminator that distinguishes style independently across image regions, enhancing discriminative capability.

Proposed method

• A multi-layer style projector (MSP) is used to encode style representations from reference glyphs, enabling hierarchical style feature learning.
• A cluster-level contrastive style (CCS) loss is proposed to maximize similarity between same-style samples and minimize it between different-style samples in the latent space.
• The generator uses an attention-based architecture to preserve global character structure while transferring style from few reference images.
• A multi-task patch discriminator is designed to evaluate both content and style at multiple spatial regions, ensuring localized and global consistency.
• The training framework combines adversarial loss, L1 loss, and the proposed contrastive style loss to optimize image quality and style fidelity.
• The model uses a memory dictionary to store class centers of style codes, enabling efficient retrieval and contrastive learning during training.

Experimental results

Research questions

• RQ1 Can modeling inter-style differences and intra-style similarities improve few-shot font generation quality and reduce artifacts?
• RQ2 How does contrastive learning on style representations enhance style consistency and reduce local distortions in generated fonts?
• RQ3 To what extent does a multi-task patch discriminator improve the discrimination of style across different image regions compared to standard discriminators?
• RQ4 Does learning style relationships rather than explicit disentanglement lead to better generalization with only a few reference samples?
• RQ5 How does the proposed cluster-level contrastive style loss compare to other loss functions in terms of style representation quality and generation fidelity?

Key findings

• DS-Font achieves the best performance on the UFSC dataset across most metrics, including the highest style classification accuracy (Acc (S)) and lowest FID score.
• On the UCSF dataset, DS-Font outperforms all baselines in FID (S) and Acc (S), demonstrating superior feature distribution alignment and style recognition.
• The model with CCS loss shows a significant improvement in style classification accuracy (up to 95% on some splits) and a 12% reduction in FID score compared to the ablated version without CCS loss.
• Qualitative results show that DS-Font successfully preserves fine-grained stroke details and global structure, while methods like LF-Font and MX-Font fail to maintain structural integrity or generate complex styles.
• The multi-task patch discriminator outperforms both standard patch and single-task discriminators, especially in content-aware metrics and visual quality, as shown in Figure 5.
• Ablation studies confirm that the CCS loss is critical for learning distinctive style representations, as its removal leads to noticeable artifacts and reduced style fidelity.