[Paper Review] Scene Text Synthesis for Efficient and Effective Deep Network Training

Changgong Zhang, Fangneng Zhan+2 more|arXiv (Cornell University)|Jan 26, 2019

Handwritten Text Recognition TechniquesComputer Science51 references33 citations

TL;DR

The paper presents a foreground–background embedding technique to synthesize annotated training images for scene text, with two components—context-aware semantic coherence and harmonious appearance adaptation—evaluated on scene text detection and recognition, achieving comparable or better performance than real images.

ABSTRACT

A large amount of annotated training images is critical for training accurate and robust deep network models but the collection of a large amount of annotated training images is often time-consuming and costly. Image synthesis alleviates this constraint by generating annotated training images automatically by machines which has attracted increasing interest in the recent deep learning research. We develop an innovative image synthesis technique that composes annotated training images by realistically embedding foreground objects of interest (OOI) into background images. The proposed technique consists of two key components that in principle boost the usefulness of the synthesized images in deep network training. The first is context-aware semantic coherence which ensures that the OOI are placed around semantically coherent regions within the background image. The second is harmonious appearance adaptation which ensures that the embedded OOI are agreeable to the surrounding background from both geometry alignment and appearance realism. The proposed technique has been evaluated over two related but very different computer vision challenges, namely, scene text detection and scene text recognition. Experiments over a number of public datasets demonstrate the effectiveness of our proposed image synthesis technique - the use of our synthesized images in deep network training is capable of achieving similar or even better scene text detection and scene text recognition performance as compared with using real images.

Motivation & Objective

• Reduce annotation cost for training deep networks by generating annotated synthetic images.
• Develop a synthesis pipeline that places foreground objects in semantically coherent contexts.
• Ensure geometric and appearance realism to improve usefulness of synthetic data for training.
• Evaluate the technique on scene text detection and recognition benchmarks to compare with real-image training.

Proposed method

• Embed foreground objects of interest into background images while preserving semantic coherence.
• Enforce context-aware placement so OOIs align with semantically meaningful regions in the background.
• Apply harmonious appearance adaptation to achieve geometric alignment and appearance realism between OOI and background.
• Produce annotated synthetic training images suitable for deep network training.
• Assess the impact of the synthesis technique on downstream scene text detection and recognition tasks.

Experimental results

Research questions

• RQ1 Can synthesized images trained with the proposed method achieve comparable text detection and recognition performance to real images?
• RQ2 Does context-aware semantic coherence improve training effectiveness for scene text tasks?
• RQ3 Does harmonious appearance adaptation enhance the realism and usefulness of embedded objects for deep learning?
• RQ4 How do synthetic images compare against real images in training robust scene text models?

Key findings

• Synthesized images using the proposed technique are effective for training deep networks.
• The approach achieves similar or better performance in scene text detection and recognition compared to using real images.
• Context-aware coherence and appearance adaptation contribute to training usefulness of synthetic data.
• Experiments validate the value of realistic foreground embedding in improving model robustness.