[Paper Review] Robust Scene Text Recognition with Automatic Rectification

Baoguang Shi, Xinggang Wang+3 more|arXiv (Cornell University)|Mar 12, 2016

Handwritten Text Recognition TechniquesComputer Science40 references88 citations

TL;DR

This paper proposes RARE, a deep learning model for robust scene text recognition that automatically rectifies irregular text using a Spatial Transformer Network (STN) with Thin-Plate-Spline (TPS) transformation, followed by an attention-based sequence recognition network (SRN). The end-to-end trainable model achieves state-of-the-art performance on benchmarks like SVT-Perspective and CUTE80, significantly improving recognition accuracy on perspective and curved text without requiring geometric ground truth for rectification.

ABSTRACT

Recognizing text in natural images is a challenging task with many unsolved problems. Different from those in documents, words in natural images often possess irregular shapes, which are caused by perspective distortion, curved character placement, etc. We propose RARE (Robust text recognizer with Automatic REctification), a recognition model that is robust to irregular text. RARE is a specially-designed deep neural network, which consists of a Spatial Transformer Network (STN) and a Sequence Recognition Network (SRN). In testing, an image is firstly rectified via a predicted Thin-Plate-Spline (TPS) transformation, into a more "readable" image for the following SRN, which recognizes text through a sequence recognition approach. We show that the model is able to recognize several types of irregular text, including perspective text and curved text. RARE is end-to-end trainable, requiring only images and associated text labels, making it convenient to train and deploy the model in practical systems. State-of-the-art or highly-competitive performance achieved on several benchmarks well demonstrates the effectiveness of the proposed model.

Motivation & Objective

• To address the challenge of recognizing irregular text in natural scenes, such as perspective and curved text, which are difficult for standard OCR systems.
• To develop a method that automatically rectifies irregular text into a more readable, regular form without requiring manual annotation of geometric transformations.
• To enable end-to-end training of a recognition system that jointly optimizes rectification and recognition using only image and text label supervision.
• To improve recognition performance on challenging benchmarks like SVT-Perspective and CUTE80, especially in lexicon-free and full-lexicon settings.

Proposed method

• The model combines a Spatial Transformer Network (STN) with a Sequence Recognition Network (SRN), where the STN applies a differentiable TPS transformation to rectify input images.
• The STN regresses fiducial point coordinates via a CNN to generate a TPS transformation that maps irregular text into a more regular, horizontal layout.
• The SRN uses an encoder-decoder architecture with an attention mechanism to recognize text as a sequence of characters from the rectified feature maps.
• The entire network is trained end-to-end using back-propagation, with gradients from the SRN's loss flowing back to supervise the STN's transformation parameters without requiring ground-truth fiducial points.
• The STN learns to predict optimal fiducial points that align text lines into a straight, readable format, improving recognition performance.
• The model leverages a convolutional-recurrent structure in the SRN encoder, enabling effective sequential feature representation for attention-based decoding.

Experimental results

Research questions

• RQ1 Can a deep learning model automatically rectify irregular scene text—such as perspective and curved text—into a form suitable for standard recognition models?
• RQ2 Can end-to-end training of a rectification and recognition system be achieved without geometric supervision, using only image and text label pairs?
• RQ3 Does the integration of a differentiable STN with an attention-based sequence recognizer improve recognition accuracy on irregular text compared to prior methods?
• RQ4 How does the model perform on benchmarks with challenging text shapes, such as SVT-Perspective and CUTE80, especially in lexicon-free settings?

Key findings

• On the SVT-Perspective dataset, RARE achieves 91.2% accuracy with a 50-word lexicon and 77.4% without, outperforming prior state-of-the-art methods.
• On the CUTE80 dataset, which focuses on curved text, RARE achieves 59.2% accuracy in the lexicon-free setting, surpassing [17] (42.7%) and [32] (54.9%).
• The model significantly improves recognition accuracy on perspective text, where the rectification mechanism alleviates the challenges posed by strong distortions.
• Qualitative results show that the STN effectively predicts fiducial points along text boundaries, producing rectified images that are more readable for the SRN.
• The model demonstrates robustness to various irregular text types, including perspective and curved text, without requiring specialized design for each type.
• Despite occasional failures on extreme distortions, the end-to-end training scheme enables the STN to learn effective rectification implicitly through recognition loss back-propagation.