[Paper Review] Character Region Awareness for Text Detection

Youngmin Baek, Bado Lee+3 more|arXiv (Cornell University)|Apr 3, 2019

Handwritten Text Recognition Techniques37 references58 citations

TL;DR

CRAFT detects individual characters using region and affinity maps with a weakly supervised training strategy, achieving state-of-the-art results on multiple curved/text-arbitrary datasets without fine-tuning.

ABSTRACT

Scene text detection methods based on neural networks have emerged recently and have shown promising results. Previous methods trained with rigid word-level bounding boxes exhibit limitations in representing the text region in an arbitrary shape. In this paper, we propose a new scene text detection method to effectively detect text area by exploring each character and affinity between characters. To overcome the lack of individual character level annotations, our proposed framework exploits both the given character-level annotations for synthetic images and the estimated character-level ground-truths for real images acquired by the learned interim model. In order to estimate affinity between characters, the network is trained with the newly proposed representation for affinity. Extensive experiments on six benchmarks, including the TotalText and CTW-1500 datasets which contain highly curved texts in natural images, demonstrate that our character-level text detection significantly outperforms the state-of-the-art detectors. According to the results, our proposed method guarantees high flexibility in detecting complicated scene text images, such as arbitrarily-oriented, curved, or deformed texts.

Motivation & Objective

• Motivate robust detection of text with arbitrary shapes beyond word-level boxes.
• Propose a character-centric detection framework that localizes character regions and links them into text instances.
• Develop a weakly supervised ground-truth generation mechanism to train on real data lacking character annotations.
• Demonstrate state-of-the-art performance on datasets with curved and polygonal text.
• Show robustness to scale variance by localizing characters instead of entire text instances.

Proposed method

• A fully convolutional backbone (VGG-16 with skip connections) outputs two heatmaps: character region scores and character affinity scores.
• Ground-truth generation uses Gaussian heatmaps warped to character boxes to create region and affinity labels.
• Weakly supervised learning generates pseudo-ground-truths for real images by dividing word-level annotations into character-level boxes via watershed, weighted by a confidence map.
• Training combines synthetic (SynthText) data and real-world word-level data, with confidence-weighted losses to cope with imperfect pseudo-GTs.
• Inference produces word-level quadrilateral boxes by thresholding the region and affinity maps and applying connected component labeling, with optional polygon-style post-processing for curved text.

Experimental results

Research questions

• RQ1 Can a character-centric representation outperform word-centric detectors on texts with irregular shapes (curved, long, or arbitrarily deformed)?
• RQ2 Does weakly supervised learning enable effective character-level detection without explicit character annotations on real data?
• RQ3 How does character-level linking via affinity maps affect robustness to scale, orientation, and perspective distortion?
• RQ4 Can the model generalize across datasets with different annotation styles (rectangles, quadrilaterals, polygons) without extensive fine-tuning?

Key findings

• CRAFT achieves state-of-the-art performance on quadrilateral datasets (ICDAR 2013/2015/2017, MSRA-TD500) with strong H-mean scores (e.g., IC13: 95.2 H; IC15: 86.9 H; IC17: 73.9 H) and a reported 8.6 FPS on IC13.
• On polygon-type datasets (TotalText, CTW-1500), CRAFT attains high H-measures (TotalText: 83.6; CTW-1500: 83.5).
• Compared to end-to-end detectors, CRAFT delivers competitive H-means (TotalText: 83.6; CTW-1500: 83.5) despite not being trained end-to-end for recognition.
• The method demonstrates robust performance without multi-scale testing, leveraging small receptive fields that focus on intra- and inter-character relations rather than whole text instances.
• A supplementary LinkRefiner improves handling of CTW-1500-style polygon annotations by refining affinities for long curved texts, achieving better polygonal results.