[Paper Review] Synthetic-to-Real Unsupervised Domain Adaptation for Scene Text Detection in the Wild

Weijia Wu, Lu Ning+1 more|arXiv (Cornell University)|Sep 3, 2020

Handwritten Text Recognition Techniques46 references4 citations

TL;DR

This paper proposes a synthetic-to-real unsupervised domain adaptation method for scene text detection using text self-training (TST) and adversarial text instance alignment (ATA). TST reduces false positives and negatives from pseudo-labels, while ATA enforces domain-invariant features via adversarial learning, achieving up to 11.3% F-score improvement on ICDAR2013 and 9.3% on ICDAR2015 when transferring from synthetic to real data.

ABSTRACT

Deep learning-based scene text detection can achieve preferable performance, powered with sufficient labeled training data. However, manual labeling is time consuming and laborious. At the extreme, the corresponding annotated data are unavailable. Exploiting synthetic data is a very promising solution except for domain distribution mismatches between synthetic datasets and real datasets. To address the severe domain distribution mismatch, we propose a synthetic-to-real domain adaptation method for scene text detection, which transfers knowledge from synthetic data (source domain) to real data (target domain). In this paper, a text self-training (TST) method and adversarial text instance alignment (ATA) for domain adaptive scene text detection are introduced. ATA helps the network learn domain-invariant features by training a domain classifier in an adversarial manner. TST diminishes the adverse effects of false positives~(FPs) and false negatives~(FNs) from inaccurate pseudo-labels. Two components have positive effects on improving the performance of scene text detectors when adapting from synthetic-to-real scenes. We evaluate the proposed method by transferring from SynthText, VISD to ICDAR2015, ICDAR2013. The results demonstrate the effectiveness of the proposed method with up to 10% improvement, which has important exploration significance for domain adaptive scene text detection. Code is available at https://github.com/weijiawu/SyntoReal_STD

Motivation & Objective

• To address the domain shift between synthetic and real scene text datasets, which degrades model performance when trained solely on synthetic data.
• To reduce the negative impact of noisy pseudo-labels (false positives and false negatives) in self-training for scene text detection.
• To learn domain-invariant features for scene text detectors using adversarial training on text instance-level representations.
• To enable effective knowledge transfer from synthetic data (source domain) to real-world data (target domain) without requiring labeled real data.

Proposed method

• Text self-training (TST) is introduced to refine pseudo-labels by filtering low-confidence predictions using stroke width consistency and confidence thresholds.
• A threshold-based filtering mechanism with parameters ε₁ and ε₂ is applied to suppress false positives and false negatives in pseudo-labeled instances.
• Adversarial text instance alignment (ATA) trains a domain discriminator to distinguish between source (synthetic) and target (real) text features in an adversarial manner.
• The model is jointly optimized using a detection head, TST refinement, and adversarial loss to align features across domains while preserving detection accuracy.
• The method is evaluated on EAST-based detectors, leveraging unlabeled real data during fine-tuning after pretraining on synthetic data.
• The framework is trained end-to-end with a combination of supervised loss on source data and self-training with refined pseudo-labels on target data.

Experimental results

Research questions

• RQ1 Can text self-training effectively reduce false positives and false negatives in pseudo-labels during domain adaptation for scene text detection?
• RQ2 To what extent can adversarial text instance alignment improve feature generalization across synthetic and real domains?
• RQ3 How effective is the proposed method in transferring knowledge from synthetic data to real-world scene text detection benchmarks?
• RQ4 What is the impact of hyperparameter choices (e.g., ε₁, ε₂) on the performance of the self-training component?

Key findings

• The proposed method achieved a 11.3% absolute improvement in F-score when transferring from SynthText to ICDAR2013, significantly outperforming the baseline.
• On the SynthText → ICDAR2015 transfer, the method improved F-score by 9.3%, demonstrating strong domain adaptation performance.
• When using VISD as the source domain, the method achieved a 9.3% F-score gain on ICDAR2015, confirming its effectiveness across different synthetic data distributions.
• The ablation study confirmed that combining both TST and ATA yields the highest performance gain (7.5% on SynthText → ICDAR2015), with each component contributing independently.
• Parameter sensitivity analysis showed that ε₁ = 3 and ε₂ = 0.3 yielded optimal performance, with results relatively robust to hyperparameter variations.
• Qualitative results demonstrated improved robustness to complex backgrounds and diverse text shapes, with reduced false positives and better localization.