[Paper Review] Multi-Oriented Scene Text Detection via Corner Localization and Region Segmentation

Pengyuan Lyu, Cong Yao+3 more|arXiv (Cornell University)|Feb 25, 2018

Handwritten Text Recognition Techniques37 references42 citations

TL;DR

A unified CNN-based detector that localizes text via corner points and scores candidate boxes with position-sensitive segmentation, enabling robust detection of arbitrarily oriented, long, and multi-script text without heavy post-processing.

ABSTRACT

Previous deep learning based state-of-the-art scene text detection methods can be roughly classified into two categories. The first category treats scene text as a type of general objects and follows general object detection paradigm to localize scene text by regressing the text box locations, but troubled by the arbitrary-orientation and large aspect ratios of scene text. The second one segments text regions directly, but mostly needs complex post processing. In this paper, we present a method that combines the ideas of the two types of methods while avoiding their shortcomings. We propose to detect scene text by localizing corner points of text bounding boxes and segmenting text regions in relative positions. In inference stage, candidate boxes are generated by sampling and grouping corner points, which are further scored by segmentation maps and suppressed by NMS. Compared with previous methods, our method can handle long oriented text naturally and doesn't need complex post processing. The experiments on ICDAR2013, ICDAR2015, MSRA-TD500, MLT and COCO-Text demonstrate that the proposed algorithm achieves better or comparable results in both accuracy and efficiency. Based on VGG16, it achieves an F-measure of 84.3% on ICDAR2015 and 81.5% on MSRA-TD500.

Motivation & Objective

• Motivate robust detection of scene text with arbitrary orientation and varying aspect ratios.
• Combine corner-point localization with region-based segmentation to improve accuracy and efficiency.
• Avoid heavy post-processing by scoring candidate boxes with segmentation maps.
• Enable end-to-end training for joint optimization of detection and segmentation tasks.

Proposed method

• Detect corner points (top-left, top-right, bottom-right, bottom-left) via default boxes across a multi-scale CNN backbone.
• Generate rotated text proposals by sampling and grouping detected corner points into candidate bounding boxes.
• Score candidate boxes using position-sensitive segmentation maps and Rotated Position-Sensitive ROI Average pooling.
• Train with a joint loss over corner point detection, localization, and segmentation (L = L_conf/N_c + λ1 L_loc/N_c + λ2 L_seg/N_s).
• Employ a Rotated Position-Sensitive ROI pooling module to handle arbitrary orientations during scoring.
• Apply end-to-end training on SynthText pretraining followed by fine-tuning on target datasets.

Experimental results

Research questions

• RQ1 Can corner-point localization coupled with region-based segmentation outperform direct regression or pure segmentation methods for multi-oriented text?
• RQ2 How does the proposed Rotated Position-Sensitive ROI pooling affect scoring of rotated text proposals?
• RQ3 What is the impact of joint optimization of corner detection and segmentation on accuracy and speed across benchmarks?
• RQ4 Is the method scalable to long text lines and multi-script text without heavy post-processing?

Key findings

• F-measure of 84.3% on ICDAR2015, 81.5% on MSRA-TD500, and 72.4% on MLT using the proposed method.
• Single-scale ICDAR2015 results: F-measure 80.7% with our method (multi-scale 84.3%).
• Long oriented text and multi-lingual text detection achieve state-of-the-art performance on MSRA-TD500 and competitive results on MLT and COCO-Text.
• The method processes over 10.4 images per second (512x512) on a Titan X GPU, indicating competitive efficiency.
• A baseline regression-based detector underperforms the corner-point method (e.g., 53.3 F-measure vs 80.7 on ICDAR2015).
• The approach shows strong generalization, achieving 42.5% F-measure on COCO-Text without training on COCO-Text.