[Paper Review] Efficient Scene Text Localization and Recognition with Local Character Refinement

Lukáš Neumann, Jiřı́ Matas|arXiv (Cornell University)|Apr 14, 2015

Handwritten Text Recognition Techniques19 references4 citations

TL;DR

This paper proposes an end-to-end real-time scene text localization and recognition method that uses MSERs with a novel stroke area ratio feature based on Stroke Support Pixels (SSPs) to detect text fragments regardless of character count or orientation. It improves robustness by refining initial hypotheses with a local segmentation model, achieving state-of-the-art f-measure of 77.6% on the ICDAR 2013 dataset with 800ms per image average runtime.

ABSTRACT

An unconstrained end-to-end text localization and recognition method is presented. The method detects initial text hypothesis in a single pass by an efficient region-based method and subsequently refines the text hypothesis using a more robust local text model, which deviates from the common assumption of region-based methods that all characters are detected as connected components. Additionally, a novel feature based on character stroke area estimation is introduced. The feature is efficiently computed from a region distance map, it is invariant to scaling and rotations and allows to efficiently detect text regions regardless of what portion of text they capture. The method runs in real time and achieves state-of-the-art text localization and recognition results on the ICDAR 2013 Robust Reading dataset.

Motivation & Objective

• To address the limitations of region-based text detection methods that assume characters are single connected components, which are sensitive to noise and fragmentation.
• To improve text localization accuracy by detecting arbitrary text fragments—single characters, multi-character groups, or partial words—without relying on strict connected component assumptions.
• To develop a robust, scalable feature for text region detection that is invariant to scale, rotation, and noise, enabling efficient detection of partial or fragmented text.
• To enhance recognition performance by combining iterative local refinement with a GMM-based color model trained on SSPs, improving segmentation quality.

Proposed method

• Initial text hypotheses are generated using MSERs enhanced with a novel stroke area ratio feature ($\varsigma$) that estimates the proportion of region pixels belonging to character strokes.
• The stroke area ratio feature is computed efficiently from a region distance map, making it invariant to scale and rotation and more robust to noise than single-stroke-width estimation methods.
• Text lines are formed by grouping MSERs using a baseline estimation model, followed by iterative refinement using a local segmentation model that classifies pixels as foreground, background, or ignored.
• Stroke Support Pixels (SSPs) are identified as central points along character strokes, used both to compute the stroke area ratio and to train a more accurate GMM-based color model for segmentation.
• A graph-based OCR module with second-order language model features is used to resolve ambiguities and generate the final word sequence from segmented components.
• The entire pipeline runs in multiple scales, with overlapping detections merged by selecting the lowest-cost path in the graph, ensuring robust end-to-end recognition.

Experimental results

Research questions

• RQ1 Can a unified feature be developed to detect text regions regardless of whether they represent single characters, multi-character groups, or partial words?
• RQ2 How can the robustness of region-based text detection be improved by moving beyond the assumption that characters must be single connected components?
• RQ3 To what extent can a local segmentation model based on Stroke Support Pixels improve text line refinement and reduce false positives in complex scenes?
• RQ4 Can a feature based on stroke area estimation from distance maps achieve invariance to scale and rotation while remaining computationally efficient?
• RQ5 How does the integration of SSPs into the color model improve segmentation accuracy in low-contrast or noisy regions?

Key findings

• The method achieves a state-of-the-art f-measure of 77.6% on the ICDAR 2013 Robust Reading dataset, outperforming previous methods including the 2013 competition winner.
• The method reports a recall of 72.4% and precision of 81.8%, significantly improving over the 2013 winner’s 66% recall and 76.2% f-measure.
• End-to-end recognition correctly localizes and recognizes 45% of words (543 out of 1189), with only 79 hallucinated words not overlapping with ground truth.
• The average runtime is 800ms per image on a standard 2.7GHz PC, demonstrating real-time feasibility.
• The method is particularly effective at detecting fragmented or partially visible text, as the SSP-based feature detects text regions regardless of the number of characters they contain.
• Failure cases are primarily due to character-like objects (e.g. arrows, pictographs) and low-contrast text not captured in the initial MSER stage.