[Paper Review] Separating Content from Style Using Adversarial Learning for Recognizing Text in the Wild

Canjie Luo, Qingxiang Lin+3 more|arXiv (Cornell University)|Jan 13, 2020

Handwritten Text Recognition Techniques57 references4 citations

TL;DR

This paper proposes a novel adversarial learning framework that separates text content from complex backgrounds in scene images to improve text recognition accuracy. By leveraging an attention-based recognizer and a generative adversarial network with character-level adversarial training and a feedback mechanism, the method achieves state-of-the-art performance using only unpaired images and text labels, significantly reducing recognition difficulty without requiring pixel-level annotations or paired data.

ABSTRACT

We propose to improve text recognition from a new perspective by separating the text content from complex backgrounds. As vanilla GANs are not sufficiently robust to generate sequence-like characters in natural images, we propose an adversarial learning framework for the generation and recognition of multiple characters in an image. The proposed framework consists of an attention-based recognizer and a generative adversarial architecture. Furthermore, to tackle the issue of lacking paired training samples, we design an interactive joint training scheme, which shares attention masks from the recognizer to the discriminator, and enables the discriminator to extract the features of each character for further adversarial training. Benefiting from the character-level adversarial training, our framework requires only unpaired simple data for style supervision. Each target style sample containing only one randomly chosen character can be simply synthesized online during the training. This is significant as the training does not require costly paired samples or character-level annotations. Thus, only the input images and corresponding text labels are needed. In addition to the style normalization of the backgrounds, we refine character patterns to ease the recognition task. A feedback mechanism is proposed to bridge the gap between the discriminator and the recognizer. Therefore, the discriminator can guide the generator according to the confusion of the recognizer, so that the generated patterns are clearer for recognition. Experiments on various benchmarks, including both regular and irregular text, demonstrate that our method significantly reduces the difficulty of recognition. Our framework can be integrated into recent recognition methods to achieve new state-of-the-art recognition accuracy.

Motivation & Objective

• To reduce the difficulty of scene text recognition caused by complex backgrounds, arbitrary shapes, and font variations.
• To address the lack of paired training data for image-to-image translation in text image generation.
• To improve recognition accuracy by normalizing background styles while preserving text content at the character level.
• To enable end-to-end training using only input images and corresponding text labels, avoiding costly pixel-wise annotations.
• To develop a feedback mechanism between discriminator and recognizer that enhances generation quality through recognition confusion.

Proposed method

• The framework employs an attention-based recognizer to extract character-level features and guide adversarial training.
• A generative adversarial network (GAN) is trained to remove complex backgrounds while preserving text content, using only unpaired real images and text labels.
• An interactive joint training scheme shares attention masks from the recognizer to the discriminator, enabling character-level supervision.
• The discriminator is guided by the recognizer's confusion, refining generated images to produce clearer, more recognizable character patterns.
• Target style samples are synthesized online during training by randomly selecting single characters, eliminating the need for large-scale paired data.
• A feedback mechanism bridges the recognizer and discriminator, allowing the generator to produce images that are more robust to recognition errors.

Experimental results

Research questions

• RQ1 Can adversarial learning be effectively adapted to separate text content from complex backgrounds in scene images without requiring paired training data?
• RQ2 How can character-level adversarial supervision improve the fidelity of generated text images compared to global, coarse-grained GANs?
• RQ3 Can a feedback mechanism between the recognizer and discriminator enhance the clarity and recognizability of generated text patterns?
• RQ4 To what extent does background style normalization improve recognition accuracy on real-world benchmarks?
• RQ5 Can the proposed method be integrated into existing recognition frameworks to achieve state-of-the-art performance with minimal modification?

Key findings

• The proposed method achieves new state-of-the-art recognition accuracy on multiple benchmarks, including a 6.0% relative improvement on the SVT-P dataset.
• Even when trained solely on synthetic data, the method outperforms a baseline trained on real data, demonstrating the effectiveness of background style normalization.
• The integration of real data (50k samples) with synthetic data further boosts performance, indicating that the method generalizes well and reduces domain gap.
• The use of online synthesized target style samples (single characters) enables effective training without requiring large-scale paired datasets.
• The feedback mechanism between the discriminator and recognizer leads to clearer character patterns, reducing recognition confusion and improving robustness.
• Experiments show that RGB space generation does not outperform grayscale, indicating that the key challenge lies in fine-grained supervision, not color representation.