[Paper Review] Scene Text Detection via Holistic, Multi-Channel Prediction

Cong Yao, Xiang Bai+4 more|arXiv (Cornell University)|Jun 29, 2016

Handwritten Text Recognition Techniques52 references212 citations

TL;DR

The paper treats scene text detection as semantic segmentation and jointly predicts text regions, characters, and linking orientations with a single FCN, enabling multi-oriented and curved text detection and achieving state-of-the-art results on ICDAR 2013/2015, MSRA-TD500, and COCO-Text.

ABSTRACT

Recently, scene text detection has become an active research topic in computer vision and document analysis, because of its great importance and significant challenge. However, vast majority of the existing methods detect text within local regions, typically through extracting character, word or line level candidates followed by candidate aggregation and false positive elimination, which potentially exclude the effect of wide-scope and long-range contextual cues in the scene. To take full advantage of the rich information available in the whole natural image, we propose to localize text in a holistic manner, by casting scene text detection as a semantic segmentation problem. The proposed algorithm directly runs on full images and produces global, pixel-wise prediction maps, in which detections are subsequently formed. To better make use of the properties of text, three types of information regarding text region, individual characters and their relationship are estimated, with a single Fully Convolutional Network (FCN) model. With such predictions of text properties, the proposed algorithm can simultaneously handle horizontal, multi-oriented and curved text in real-world natural images. The experiments on standard benchmarks, including ICDAR 2013, ICDAR 2015 and MSRA-TD500, demonstrate that the proposed algorithm substantially outperforms previous state-of-the-art approaches. Moreover, we report the first baseline result on the recently-released, large-scale dataset COCO-Text.

Motivation & Objective

• Cast scene text detection as a semantic segmentation problem to leverage global image context.
• Predict three text-related properties jointly: text regions, individual characters, and the linking orientation between characters.
• Develop a pipeline that forms detections from pixel-wise maps using segmentation, graph-based grouping, and partitioning.
• Demonstrate robustness to multi-oriented and curved text and validate on standard benchmarks (ICDAR 2013/2015, MSRA-TD500) and COCO-Text.

Proposed method

• Extend a Fully Convolutional Network (FCN) inspired by HED to output three prediction maps per image: text regions, characters (shrunk during training), and linking orientations.
• Ground truth maps consist of a binary region map, a binary character map, and a soft orientation map; orientation is defined in the range [−π/2, π/2] and normalized to [0,1].
• Fuse and train using a multi-channel loss that combines text-region, character, and orientation losses in a weighted objective with equal weights (λ1=λ2=λ3=1/3).
• During inference, generate prediction maps, apply adaptive thresholding to obtain text regions and character candidates, and use Delaunay triangulation plus a graph to link characters into text lines.
• Graph-based grouping uses a maximum spanning tree and a straightness/distance/orientation-based scoring to partition characters into text lines, accommodating curved text with a threshold τ to handle non-linear layouts.
• Performance is fused across scales during testing to produce final detections.

Experimental results

Research questions

• RQ1 Can scene text detection be improved by moving from local region-based decisions to holistic, pixel-level predictions across the whole image?
• RQ2 Does predicting additional text properties (characters and linking orientations) within a single FCN improve the separation and grouping of adjacent text instances?
• RQ3 Can a graph-based, multi-channel prediction framework robustly detect multi-oriented and curved text in natural scenes?
• RQ4 How does holistic multi-channel text detection perform on standard benchmarks (ICDAR 2013/2015, MSRA-TD500) and COCO-Text compared to prior art?

Key findings

• The proposed method achieves high recall on ICDAR 2013 (0.8022) with precision 0.8888 and F-measure 0.8433.
• On ICDAR 2015, the method yields precision 0.7226, recall 0.5869, and F-measure 0.6477, outperforming several baselines in recall and close to the best in precision.
• On MSRA-TD500, the method achieves precision 0.7651, recall 0.7531, and F-measure 0.7591, with notable improvement in recall over prior work.
• On COCO-Text validation, the method reports precision 0.4323, recall 0.271, and F-measure 0.3331, demonstrating scalability to a large, diverse dataset.
• The approach shows qualitative robustness to various languages, scripts, curved text, and challenging real-world conditions.
• Inference runs at about 0.42s per 640x480 image on a K40m GPU, with ~0.2s CPU post-processing.