[Paper Review] UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World

Shangbang Long, Cong Yao|arXiv (Cornell University)|Mar 24, 2020

Handwritten Text Recognition Techniques52 references44 citations

TL;DR

UnrealText renders photo-realistic scene text images from 3D Unreal Engine worlds to improve scene text detection and recognition, offering large multilingual datasets and full-symbol annotations.

ABSTRACT

Synthetic data has been a critical tool for training scene text detection and recognition models. On the one hand, synthetic word images have proven to be a successful substitute for real images in training scene text recognizers. On the other hand, however, scene text detectors still heavily rely on a large amount of manually annotated real-world images, which are expensive. In this paper, we introduce UnrealText, an efficient image synthesis method that renders realistic images via a 3D graphics engine. 3D synthetic engine provides realistic appearance by rendering scene and text as a whole, and allows for better text region proposals with access to precise scene information, e.g. normal and even object meshes. The comprehensive experiments verify its effectiveness on both scene text detection and recognition. We also generate a multilingual version for future research into multilingual scene text detection and recognition. Additionally, we re-annotate scene text recognition datasets in a case-sensitive way and include punctuation marks for more comprehensive evaluations. The code and the generated datasets are released at: https://github.com/Jyouhou/UnrealText/ .

Motivation & Objective

• Motivate the need for scalable, realistic synthetic data for scene text detection and recognition beyond background-embedded text methods.
• Propose a 3D world-based text synthesis engine (UnrealText) to produce labeled data with rich scene information.
• Show that 3D rendering improves detector/recognizer performance and enable multilingual datasets with comprehensive annotations.
• Release large-scale English and multilingual synthetic datasets and re-annotate existing benchmarks for full-symbol evaluation.

Proposed method

• Render text as planar mesh textures within UE4-based scenes to ensure joint scene-text realism (illumination, occlusion, perspective).
• Develop a viewfinding module using a physically constrained 3D random walk with auxiliary camera anchors to generate diverse viewpoints.
• Incorporate environment randomization by varying lighting, fog, and other conditions to simulate real-world variation.
• Propose a two-stage text region generation pipeline: initial proposals from surface normal maps, then 3D refinement on object meshes for natural perspective distortion.
• Render text by triangulating refined proposals into planar meshes, applying textures with varied font/colors, and collecting corresponding ground-truth content.
• Achieve efficient rendering (0.7–1.5s per image) and generate large multilingual datasets using UE4 and UnrealCV.

Experimental results

Research questions

• RQ1 Can 3D scene-based text synthesis outperform previous 2D-background text embedding methods for detector/recognizer training?
• RQ2 How do viewfinding and environment randomization affect synthetic data diversity and downstream performance?
• RQ3 What benefits do 3D-augmented, mesh-based text proposals provide over traditional region proposals in scene text synthesis?
• RQ4 How effective is UnrealText in generating multilingual and richly annotated datasets for scene text recognition?
• RQ5 What is the impact of pretraining with UnrealText (and combos with real data) on real-world benchmarks?

Key findings

• UnrealText-trained detectors achieve higher F1 on IC15, IC13, and MLT17 compared with prior synthetic data (e.g., 10K UnrealText vs 10K SynthText/VISD/SynthText3D).
• Combining UnrealText with VISD or SynthText improves detection performance beyond either alone, demonstrating complementarity with real-background synthetic data.
• Pretraining on UnrealText (full dataset) followed by finetuning on real data yields substantial gains across IC15, IC13, and MLT 2017, surpassing several recent state-of-the-art results in some settings.
• Mask-RCNN experiments show UnrealText and SynthText3D outperform real-background synthetic datasets, with full UnrealText plus SynthText yielding strong cross-method improvements.
• Ablation shows the viewfinding and environment randomization components significantly boost diversity and performance, especially with smaller training sets; random walk plus manual anchors offers robust scalability.