[Paper Review] Efficient and Accurate Arbitrary-Shaped Text Detection with Pixel Aggregation Network

Wenhai Wang, Enze Xie+6 more|arXiv (Cornell University)|Aug 16, 2019

Handwritten Text Recognition TechniquesComputer Science56 references75 citations

TL;DR

PAN detects arbitrary-shaped text with a lightweight segmentation head and a learnable Pixel Aggregation post-processing, achieving strong accuracy with real-time to near-real-time speed on curved-text benchmarks.

ABSTRACT

Scene text detection, an important step of scene text reading systems, has witnessed rapid development with convolutional neural networks. Nonetheless, two main challenges still exist and hamper its deployment to real-world applications. The first problem is the trade-off between speed and accuracy. The second one is to model the arbitrary-shaped text instance. Recently, some methods have been proposed to tackle arbitrary-shaped text detection, but they rarely take the speed of the entire pipeline into consideration, which may fall short in practical applications.In this paper, we propose an efficient and accurate arbitrary-shaped text detector, termed Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and a learnable post-processing. More specifically, the segmentation head is made up of Feature Pyramid Enhancement Module (FPEM) and Feature Fusion Module (FFM). FPEM is a cascadable U-shaped module, which can introduce multi-level information to guide the better segmentation. FFM can gather the features given by the FPEMs of different depths into a final feature for segmentation. The learnable post-processing is implemented by Pixel Aggregation (PA), which can precisely aggregate text pixels by predicted similarity vectors. Experiments on several standard benchmarks validate the superiority of the proposed PAN. It is worth noting that our method can achieve a competitive F-measure of 79.9% at 84.2 FPS on CTW1500.

Motivation & Objective

• Address the speed-accuracy trade-off in arbitrary-shaped scene text detection.
• Develop a lightweight segmentation head to enhance multi-scale features.
• Introduce Pixel Aggregation to merge text pixels to kernels using learned similarity.
• Enable end-to-end, efficient post-processing to reconstruct complete text instances.
• Demonstrate state-of-the-art performance on curved-text benchmarks with real-time speeds.

Proposed method

• Use ResNet-18 as a lightweight backbone for segmentation.
• Introduce a cascaded Feature Pyramid Enhancement Module (FPEM) to enlarge receptive fields at low cost.
• Use a Feature Fusion Module (FFM) to fuse multi-depth features into a final segmentation feature.
• Predict text regions, kernels, and similarity vectors for each pixel.
• Apply Pixel Aggregation (PA) to guide text pixels to their corresponding kernels using learned similarity vectors.
• Train with a combination of text/kernels losses and Pixel Aggregation losses (L_agg, L_dis) and use dice loss for segmentation.

Experimental results

Research questions

• RQ1 Can a lightweight segmentation head (FPEM + FFM) close the performance gap while maintaining high speed for arbitrary-shaped text detection?
• RQ2 Does Pixel Aggregation enable accurate reconstruction of complete text instances from kernels in real time?
• RQ3 What is the impact of PA and the cascade depth of FPEMs on accuracy and throughput across curved-text and multi-oriented benchmarks?
• RQ4 How does PAN compare to state-of-the-art methods on CTW1500, Total-Text, and other benchmarks in terms of F-measure and FPS?

Key findings

• PAN achieves competitive to state-of-the-art F-measures on curved-text benchmarks (CTW1500 and Total-Text) while delivering high FPS (e.g., PAN-320 around 84.2 FPS on CTW1500 without external pre-training; PAN-640 around 39.8 FPS on CTW1500).
• A cascaded FPEM improves feature representation with minimal additional cost, and two cascaded FPEMs provide a favorable speed/accuracy balance.
• FFM effectively fuses multi-depth features with low overhead, outperforming simple concatenation in accuracy with similar speed.
• Pixel Aggregation (PA) improves accuracy by aligning text pixels to kernels via learned similarity vectors, with ablations showing meaningful gains when PA is removed.
• Using SynthText pre-training further boosts performance (e.g., PAN-320 F around 79.9% on CTW1500; PAN-640 achieving up to 85.0% F on Total-Text).
• PAN demonstrates strong performance on curved text while maintaining real-time or near-real-time speeds, outperforming several baselines in both accuracy and speed on CTW1500, Total-Text, ICDAR 2015, and MSRA-TD500.