[Paper Review] Incidental Scene Text Understanding: Recent Progresses on ICDAR 2015 Robust Reading Competition Challenge 4

Cong Yao, Jianan Wu+5 more|arXiv (Cornell University)|Nov 30, 2015

Handwritten Text Recognition Techniques30 references17 citations

TL;DR

This paper presents a deep learning-based approach for incidental scene text understanding, framing text detection as semantic segmentation using Fully Convolutional Networks (FCN), combining CNN-LSTM with Connectionist Temporal Classification (CTC) for word recognition, and integrating both for end-to-end recognition. The method achieves state-of-the-art performance on ICDAR 2015 Robust Reading Competition Challenge 4, with an F-measure of 0.6376 on text localization, 0.6399 on word recognition, and 0.4674 on end-to-end recognition under strong contextual conditions.

ABSTRACT

Different from focused texts present in natural images, which are captured with user's intention and intervention, incidental texts usually exhibit much more diversity, variability and complexity, thus posing significant difficulties and challenges for scene text detection and recognition algorithms. The ICDAR 2015 Robust Reading Competition Challenge 4 was launched to assess the performance of existing scene text detection and recognition methods on incidental texts as well as to stimulate novel ideas and solutions. This report is dedicated to briefly introduce our strategies for this challenging problem and compare them with prior arts in this field.

Motivation & Objective

• Address the challenge of detecting and recognizing incidental scene text, which is inherently more complex than focused text due to blur, skew, poor lighting, and clutter.
• Evaluate existing methods on the ICDAR 2015 Robust Reading Competition Challenge 4, which focuses on real-world, unconstrained natural images.
• Develop and validate novel strategies for text detection, word recognition, and end-to-end recognition tailored to incidental scene text.
• Demonstrate the superiority of holistic, deep learning-based approaches over traditional, localized detection methods on complex, real-world data.

Proposed method

• Formulate scene text detection as a semantic segmentation problem using a Fully Convolutional Network (FCN) to predict per-pixel probabilities of text regions.
• Apply thresholding and connected component analysis to generate final text bounding boxes from the FCN prediction map.
• Implement a hybrid recognition model combining convolutional layers, Long Short-Term Memory (LSTM) recurrent networks, and a Connectionist Temporal Classification (CTC) layer for sequence modeling.
• Integrate a dictionary-based error correction module to refine the initial recognition output and improve word-level accuracy.
• Combine the proposed detection and recognition models into an end-to-end system for joint optimization and inference.
• Train and evaluate models on the ICDAR 2015 dataset, using standard metrics including F-measure, Total Edit Distance (TED), and Correct Recognition Word Rate (C.R.W.).

Experimental results

Research questions

• RQ1 How does a holistic, FCN-based semantic segmentation approach compare to traditional localized candidate generation for text detection in incidental scenes?
• RQ2 To what extent can a CNN-LSTM-CTC architecture with dictionary-based error correction improve word recognition accuracy on challenging incidental scene text?
• RQ3 Can the integration of detection and recognition into an end-to-end system yield better performance than separate pipelines on real-world, unconstrained data?
• RQ4 How do the proposed methods perform across different contextual settings (strong, weak, generic) in the ICDAR 2015 benchmark?
• RQ5 What is the relative improvement of the proposed method over prior state-of-the-art approaches on the ICDAR 2015 Challenge 4 and Challenge 1 datasets?

Key findings

• The proposed text detection method (Megvii-Image++) achieved an F-measure of 0.6376 on the ICDAR 2015 Text Localization task, significantly outperforming the previous SOTA (0.4984).
• For word recognition, the method reduced Total Edit Distance (TED) to 377.9 and increased Correct Recognition Word Rate (C.R.W.) to 0.6399, substantially improving over prior methods.
• On the end-to-end recognition task under the Strongly Contextualised setting, the method achieved an F-measure of 0.4674, the highest among all competing methods.
• In the more realistic Weakly Contextualised and Generic settings, the method nearly doubled all metrics, achieving F-measures of 0.400 and 0.3286, respectively.
• The system maintained state-of-the-art performance on the ICDAR 2015 Challenge 1 (Born-Digital) dataset, with an F-measure of 0.8535 under the Strong setting.
• The method demonstrated robustness and generalization, achieving high performance across diverse and challenging real-world scene text scenarios.