[Paper Review] Self-Training for Domain Adaptive Scene Text Detection

Yudi Chen, Wei Wang+4 more|arXiv (Cornell University)|May 23, 2020

Handwritten Text Recognition Techniques38 references4 citations

TL;DR

This paper proposes a self-training framework for domain-adaptive scene text detection that leverages unannotated videos and images to mine high-quality pseudo-labeled hard examples. By fusing detection and tracking results through a novel Text Mining Module (TMM) and using an image-to-video generation method (Gen-Loop) when videos are unavailable, the approach achieves state-of-the-art performance on ICDAR2015 and MSRA-TD500 with minimal annotation cost.

ABSTRACT

Though deep learning based scene text detection has achieved great progress, well-trained detectors suffer from severe performance degradation for different domains. In general, a tremendous amount of data is indispensable to train the detector in the target domain. However, data collection and annotation are expensive and time-consuming. To address this problem, we propose a self-training framework to automatically mine hard examples with pseudo-labels from unannotated videos or images. To reduce the noise of hard examples, a novel text mining module is implemented based on the fusion of detection and tracking results. Then, an image-to-video generation method is designed for the tasks that videos are unavailable and only images can be used. Experimental results on standard benchmarks, including ICDAR2015, MSRA-TD500, ICDAR2017 MLT, demonstrate the effectiveness of our self-training method. The simple Mask R-CNN adapted with self-training and fine-tuned on real data can achieve comparable or even superior results with the state-of-the-art methods.

Motivation & Objective

• To address the severe performance degradation of deep learning-based scene text detectors when deployed in new domains with limited annotated data.
• To reduce reliance on expensive and time-consuming data annotation by leveraging unannotated videos and images for self-training.
• To improve model generalization and performance in target domains through automated, noise-reduced hard example mining.
• To enable effective self-training even when only images are available, by generating synthetic video sequences via an image-to-video generation method.

Proposed method

• A self-training framework is proposed that initializes a detector on source domain data and re-trains it using pseudo-labeled examples from unannotated target domain videos or images.
• A Text Mining Module (TMM) fuses detection and tracking results to filter noisy predictions and identify hard examples with high confidence.
• Detection results are generated by a Mask R-CNN-based detector, while tracking is performed using template matching to maintain object consistency across video frames.
• An image-to-video generation method (Gen-Loop) synthesizes video-like sequences from still images to enable self-training when real videos are unavailable.
• A balance loss is applied during re-training to stabilize learning and improve convergence on the pseudo-labeled data.
• The framework supports both video and image inputs, with iterative refinement through multiple self-training loops.

Experimental results

Research questions

• RQ1 Can self-training with pseudo-labeled examples from unannotated videos and images effectively improve scene text detection performance in a target domain with no or minimal annotations?
• RQ2 How can detection and tracking results be effectively fused to reduce noise in pseudo-labels and improve the quality of mined hard examples?
• RQ3 To what extent can an image-to-video generation method (Gen-Loop) enable self-training when only images are available in the target domain?
• RQ4 How does the proposed method compare to data generation and domain adaptation techniques in terms of performance and realism of training data?
• RQ5 Can a simple detector (e.g., Mask R-CNN) achieve state-of-the-art performance through self-training without architectural modifications?

Key findings

• The proposed self-training framework achieves a 1.2% higher F-score on ICDAR2015 (84.2 F1) compared to the best baseline when fine-tuned on real target data, surpassing previous SOTA methods.
• On MSRA-TD500, the method improves F1 from 70.0 to 79.1 when using 15VID as a source and self-training on target data, demonstrating strong domain adaptation capability.
• Even without external data, the method achieves 81.0 F1 on ICDAR2015, outperforming methods that use 800k synthetic images as extra data.
• The addition of data augmentation (AUG) boosts performance to 88.2 F1 on ICDAR2015, matching or exceeding the best reported results in the literature.
• The Gen-Loop method enables effective self-training on image-only data, achieving 84.3 F1 on MSRA-TD500 with augmented data, showing its effectiveness in data-scarce scenarios.
• The TMM significantly reduces noise in pseudo-labels, as evidenced by consistent performance gains across multiple benchmarks and domains.