[Paper Review] Scene Text Detection and Recognition: The Deep Learning Era

Shangbang Long, Xin He+1 more|arXiv (Cornell University)|Nov 10, 2018

Handwritten Text Recognition TechniquesComputer Science191 references109 citations

TL;DR

This survey synthesizes how deep learning transformed scene text detection and recognition, presenting a taxonomy of methods, datasets, benchmarks, and future trends.

ABSTRACT

With the rise and development of deep learning, computer vision has been tremendously transformed and reshaped. As an important research area in computer vision, scene text detection and recognition has been inescapably influenced by this wave of revolution, consequentially entering the era of deep learning. In recent years, the community has witnessed substantial advancements in mindset, approach and performance. This survey is aimed at summarizing and analyzing the major changes and significant progresses of scene text detection and recognition in the deep learning era. Through this article, we devote to: (1) introduce new insights and ideas; (2) highlight recent techniques and benchmarks; (3) look ahead into future trends. Specifically, we will emphasize the dramatic differences brought by deep learning and the grand challenges still remained. We expect that this review paper would serve as a reference book for researchers in this field. Related resources are also collected and compiled in our Github repository: https://github.com/Jyouhou/SceneTextPapers.

Motivation & Objective

• Summarize major changes and progress in scene text detection and recognition brought by deep learning.
• Review datasets, benchmarks, and evaluation protocols used in the field.
• Analyze current status, challenges, and potential future trends in scene text understanding.
• Provide insights and a reference resource for researchers via a compiled overview and repository.

Proposed method

• Classify methods into four categories: text detection, text recognition, end-to-end systems, and auxiliary methods.
• Describe the evolution of detection methods from multi-step pipelines to one-stage and polygon-based representations.
• Explain recognition frameworks based on CTC and encoder–decoder approaches and adaptations for irregular text with rectification.
• Discuss auxiliary techniques such as synthetic data generation and cross-dataset evaluation to bolster learning.
• Summarize datasets and evaluation protocols and provide perspective on future research directions.

Experimental results

Research questions

• RQ1 How has deep learning changed the methodology and performance of scene text detection and recognition?
• RQ2 What are the dominant architectures and representations used for detecting and recognizing text in the wild?
• RQ3 How do current methods handle irregular, curved, and multi-oriented text versus straight text?
• RQ4 What datasets, benchmarks, and auxiliary data support progress in this field, and what are their limitations?
• RQ5 What are the key open challenges and future trends in scene text detection and recognition?

Key findings

• Deep learning has transformed the field by enabling end-to-end trainable pipelines and reducing reliance on hand-crafted features.
• Detection methods evolved from multi-step, text-centric pipelines to single-stage detectors and polygon/segmentation-based representations for irregular text.
• Recognition approaches largely rely on CTC or encoder–decoder frameworks, with rectification techniques to handle curved/irregular text.
• Auxiliary technologies, especially synthetic data and cross-dataset evaluations, have accelerated progress and generalization.
• A comprehensive review of datasets and evaluation protocols accompanies an outlook on future trends and research directions.