[Paper Review] Text Perceptron: Towards End-to-End Arbitrary-Shaped Text Spotting

Qiao Liang, Sanli Tang+5 more|arXiv (Cornell University)|Feb 17, 2020

Handwritten Text Recognition Techniques40 references4 citations

TL;DR

Text Perceptron proposes an end-to-end trainable framework for arbitrary-shaped text spotting by integrating a segmentation-based detector with a differentiable Shape Transform Module (STM) that dynamically optimizes fiducial points for thin-plate spline-based rectification. The method achieves state-of-the-art performance on irregular text benchmarks like Total-Text and CTW1500, significantly outperforming prior pipelined or non-end-to-end approaches by enabling joint optimization of detection and recognition through back-propagated error signals.

ABSTRACT

Many approaches have recently been proposed to detect irregular scene text and achieved promising results. However, their localization results may not well satisfy the following text recognition part mainly because of two reasons: 1) recognizing arbitrary shaped text is still a challenging task, and 2) prevalent non-trainable pipeline strategies between text detection and text recognition will lead to suboptimal performances. To handle this incompatibility problem, in this paper we propose an end-to-end trainable text spotting approach named Text Perceptron. Concretely, Text Perceptron first employs an efficient segmentation-based text detector that learns the latent text reading order and boundary information. Then a novel Shape Transform Module (abbr. STM) is designed to transform the detected feature regions into regular morphologies without extra parameters. It unites text detection and the following recognition part into a whole framework, and helps the whole network achieve global optimization. Experiments show that our method achieves competitive performance on two standard text benchmarks, i.e., ICDAR 2013 and ICDAR 2015, and also obviously outperforms existing methods on irregular text benchmarks SCUT-CTW1500 and Total-Text.

Motivation & Objective

• To address the incompatibility between conventional pipelined text detection and recognition, which limits end-to-end optimization and robustness on irregular text.
• To overcome the limitations of fixed rectification methods and non-differentiable fiducial point generation in handling curved or multi-oriented scene text.
• To design a unified, trainable framework that jointly optimizes text detection and recognition by dynamically adjusting fiducial points based on recognition feedback.
• To improve recognition accuracy on arbitrary-shaped text by explicitly modeling reading order and boundary structure through a segmentation-based detector.
• To eliminate the need for external rectification networks and reduce computational cost by embedding rectification into the end-to-end training process.

Proposed method

• A segmentation-based text detector is designed to classify each pixel into four regions: center, head, tail, and top/bottom boundary, enabling detection of text instances with learned reading order.
• The Shape Transform Module (STM) generates and refines fiducial points on text boundaries using differentiable thin-plate splines (TPS) to transform irregular text into regular, rectified forms.
• Fiducial points are dynamically tuned via back-propagation from the recognition head, enabling end-to-end optimization of both detection and recognition components.
• The framework uses a sequence-based recognition head (e.g., CRNN) to generate character sequences from the rectified features.
• The entire system is trained end-to-end, allowing error signals from recognition to refine detection and fiducial point positions.
• The method supports variable numbers of fiducial points (from 4 to 18), with performance stabilizing at 10–14 points on irregular benchmarks.

Experimental results

Research questions

• RQ1 Can a differentiable, end-to-end framework jointly optimize text detection and recognition for arbitrary-shaped text, overcoming the limitations of pipelined approaches?
• RQ2 How does the number of fiducial points affect the performance of rectification and recognition on curved and multi-oriented text?
• RQ3 Can a segmentation-based detector with structured boundary classification improve detection of closely spaced or overlapping text instances?
• RQ4 To what extent can a differentiable shape transform module with learnable fiducial points outperform non-differentiable or fixed-geometry rectification methods?
• RQ5 Does end-to-end training with back-propagated recognition errors lead to better detection and rectification than conventional two-stage training?

Key findings

• Text Perceptron achieves a detection F-score of 87.1 on ICDAR 2015 and 85.3 on Total-Text with 18 fiducial points, demonstrating strong generalization on both regular and irregular text.
• On the irregular benchmark CTW1500, the method achieves an end-to-end F-score of 84.5 with 18 fiducial points, significantly outperforming previous state-of-the-art methods.
• The performance on Total-Text and CTW1500 improves with increasing fiducial points, stabilizing at 10–14 points, indicating that more complex shapes require higher-fidelity fiducial representation.
• With 4 fiducial points, the method achieves 87.1 F-score on ICDAR 2015, showing minimal degradation compared to higher point counts, indicating sufficient capacity for regular text.
• The failure analysis reveals that overlapping text and vertical text recognition remain challenging, particularly due to limited representation in training data and recognition model ambiguity.
• Visualization results confirm that the model effectively captures reading order and rectifies curved, perspective, and vertical texts into regular forms using learned fiducial points.