[Paper Review] PixelLink: Detecting Scene Text via Instance Segmentation

Dan Deng, Haifeng Liu+2 more|arXiv (Cornell University)|Jan 4, 2018

Handwritten Text Recognition TechniquesComputer Science22 references53 citations

TL;DR

PixelLink detects scene text by performing instance segmentation through pixel links, avoiding regression-based bounding box localization and enabling direct extraction of text bounding boxes from segmentation results.

ABSTRACT

Most state-of-the-art scene text detection algorithms are deep learning based methods that depend on bounding box regression and perform at least two kinds of predictions: text/non-text classification and location regression. Regression plays a key role in the acquisition of bounding boxes in these methods, but it is not indispensable because text/non-text prediction can also be considered as a kind of semantic segmentation that contains full location information in itself. However, text instances in scene images often lie very close to each other, making them very difficult to separate via semantic segmentation. Therefore, instance segmentation is needed to address this problem. In this paper, PixelLink, a novel scene text detection algorithm based on instance segmentation, is proposed. Text instances are first segmented out by linking pixels within the same instance together. Text bounding boxes are then extracted directly from the segmentation result without location regression. Experiments show that, compared with regression-based methods, PixelLink can achieve better or comparable performance on several benchmarks, while requiring many fewer training iterations and less training data.

Motivation & Objective

• Motivate text detection without bounding box regression by leveraging instance segmentation.
• Propose a pixel-link based network to separate closely packed text instances.
• Enable direct bounding box extraction from segmentation results and compare with regression-based methods.

Proposed method

• Two-headed CNN with shared VGG16 backbone predicting text/non-text at pixels and eight-direction pixel links.
• Pixels labeled as text/non-text; links between neighboring pixels indicate same-instance connectivity.
• Instance segmentation via positive links to form connected components representing text instances.
• Bounding boxes are extracted from CCs using minAreaRect without regression-based location prediction.
• Instance-Balanced Cross-Entropy Loss with Online Hard Example Mining for robust training.
• Post-processing includes simple geometric filtering to remove noise.

Experimental results

Research questions

• RQ1 Can text instances in natural scenes be effectively detected via instance segmentation using pixel linking without location regression?
• RQ2 Do pixel-link based methods require less data or training iterations than regression-based methods while achieving comparable or better accuracy?
• RQ3 How does PixelLink perform on standard benchmarks (IC15, IC13, TD500) compared to regression-based detectors?
• RQ4 What is the impact of network resolution, linkage thresholds, and post-filtering on detection performance?
• RQ5 Is bounding-box extraction from segmentation results sufficient for competition benchmarks?

Key findings

• PixelLink achieves competitive or superior F-scores compared with regression-based methods on IC15 while using fewer training iterations and less data.
• On IC15, PixelLink 4s reaches F=82.3 with 7.3 FPS, outperforming several regression-based baselines in accuracy.
• PixelLink 2s shows higher accuracy but slower speed than the 4s variant (F=83.7, 3.0 FPS).
• Ablations show the linking mechanism is essential; removing links drastically reduces recall and precision.
• Instance-Balance and training from scratch enable faster convergence and strong performance without ImageNet pretraining.
• On IC13, PixelLink with 2s and MS achieves F around 88.1–87.5 depending on scale, outperforming several baselines.