[Paper Review] Traditional Chinese Synthetic Datasets Verified with Labeled Data for Scene Text Recognition

Yi‐Chang Chen, Yu‐Chuan Chang+2 more|arXiv (Cornell University)|Nov 26, 2021

Handwritten Text Recognition Techniques13 references4 citations

TL;DR

This paper introduces a synthetic data generation framework for Traditional Chinese scene text recognition (STR), producing over 20 million diverse synthetic images using font, background, and text property variations. Evaluated on the newly created TC-STR 7k-word benchmark, models pre-trained on synthetic data and fine-tuned on real data achieve 89.64% accuracy, demonstrating significant performance gains with minimal human-labeled data.

ABSTRACT

Scene text recognition (STR) has been widely studied in academia and industry. Training a text recognition model often requires a large amount of labeled data, but data labeling can be difficult, expensive, or time-consuming, especially for Traditional Chinese text recognition. To the best of our knowledge, public datasets for Traditional Chinese text recognition are lacking. This paper presents a framework for a Traditional Chinese synthetic data engine which aims to improve text recognition model performance. We generated over 20 million synthetic data and collected over 7,000 manually labeled data TC-STR 7k-word as the benchmark. Experimental results show that a text recognition model can achieve much better accuracy either by training from scratch with our generated synthetic data or by further fine-tuning with TC-STR 7k-word.

Motivation & Objective

• To address the lack of public, high-quality datasets for Traditional Chinese scene text recognition (STR).
• To develop a scalable synthetic data generation framework that mimics real-world text diversity for low-cost, high-volume training data.
• To create a manually annotated real-world benchmark dataset (TC-STR 7k-word) for reliable evaluation of STR models.
• To evaluate the impact of synthetic data diversity—especially background and font variation—on model generalization and accuracy.
• To demonstrate that pre-training on synthetic data followed by fine-tuning on limited real data outperforms training from scratch on real data alone.

Proposed method

• The synthetic data engine generates scene text images using a multi-stage pipeline: word sampling from curated dictionaries and Wikipedia titles, character spacing with random insertion, and diverse font rendering with 175 distinct fonts.
• Text rendering incorporates variable font sizes, colors, stroke effects, skewing, distortion, and noise to simulate real-world variation.
• Backgrounds include both simple solid colors and complex, wild backgrounds derived from real images to enhance realism and diversity.
• The model architecture uses TPS transformation, ResNet for feature extraction, BiLSTM for sequence modeling, and CTC for sequence prediction.
• Ablation studies isolate the contribution of background, font, scene, and word diversity to model performance.
• Fine-tuning is performed using a pre-trained model on synthetic data and a subset of the real TC-STR 7k-word dataset with data augmentation.

Experimental results

Research questions

• RQ1 Can a synthetic data generation framework significantly improve the performance of scene text recognition models for Traditional Chinese with minimal real-world labeled data?
• RQ2 How do variations in background, font, text distortion, and word diversity affect the generalization and accuracy of STR models?
• RQ3 Does pre-training on synthetic data followed by fine-tuning on real data outperform training from scratch on real data?
• RQ4 To what extent does the inclusion of wild, complex backgrounds in synthetic data improve model robustness compared to simple backgrounds?
• RQ5 How effective is synthetic data in reducing the need for expensive human annotation in Traditional Chinese STR?

Key findings

• A model pre-trained on 20 million synthetic images and fine-tuned on a subset of the TC-STR 7k-word dataset achieved 89.64% accuracy, outperforming models trained from scratch on real data.
• Pre-training on synthetic data alone achieved 83.51% accuracy on the TC-STR-test set, demonstrating the quality and realism of the synthetic images.
• Background diversity had the most significant impact on performance; models trained only on simple backgrounds showed substantially lower accuracy.
• Font diversity was crucial—using only one font (Source Han Sans) reduced model generalization and accuracy.
• Word diversity also significantly affected performance; removing 90% of the word dictionary degraded recognition accuracy.
• Data augmentation during fine-tuning provided only marginal improvement, suggesting that the synthetic data already captured sufficient variation to reduce the need for additional augmentation.